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EARNED  VALUE  MANAGEMENT 


The  Power  of  Data 

New  Thinking  and  Technology 
Can  Keep  EVMS  Relevant 


Melvin  Frank 
David  Kester 
Karen  Urschel 


The  U.S.  Department  of  Energy  provides  clear  and 
compelling  evidence  that  an  automated,  data-driven 
approach  is  the  best  option  for  delivering  accurate  and  cost 
effective  EVMS  compliance  for  the  government. 

—Gary  C.  Humphreys,  Humphreys  &  Associates 
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Fourth  in  a  series  of  articles 


The  first  in  this  series  of  Earned  Value  Manage¬ 
ment  (EVM)  articles,  "EVM  System's  (EVMS) 
High  Cost— Fact  or  Fiction?,"  published  in  the 
January-February  2017  issue  of  Defense  AT&L, 
was  a  compelling  review  of  why  anecdotally 
based  perceptions  of  EVMS  implementations  are  not 
substantiated  by  the  facts  found  by  the  Joint  Space 
Cost  Council  (JSCC)  study.  The  conclusion  from  the 
JSCC  study  suggests  that  there  is  a  point  at  which  the 
government's  implementation  of  EVMS  requirements 
can  drive  costs  upward  and  beyond  the  point  of  benefit 
to  either  the  government  or  its  contractor.  However, 
the  study  also  revealed  that  EVMS,  when  properly 
maintained  and  its  data  used  optimally,  provides  a 
high-value  and  cost-effective  management  tool  that 
supports  product  delivery  through  the  genuine  stew¬ 
ardship  of  U.S.  taxpayer  dollars.  Now  perhaps  more 
than  any  other  time  we  need  to  be  reminded  that  a 
project  management  methodology  that  genuinely  em¬ 
braces  the  principles  of  EVM  is  the  best  way  to  posi¬ 
tion  the  government  for  success. 

Treat  Every  Tax  Dollar  As  If  It  Were  Your  Own 

An  EVM  methodology  and  the  associated  EVMS  requirement  are  designed  to 
ensure  that  agencies  acquire  major  systems  in  the  most  effective,  economical 
and  timely  manner.  Specifically,  Federal  Acquisition  Regulation  (FAR)  Subpart 
34.2  directs  that  an  EVMS  be  required  for  major  acquisitions  in  accordance  with 
the  Office  of  Management  and  Budget's  OMB  Circular  A-11.  The  FAR  further  states 
that  government  may  also  require  an  EVMS  for  other  acquisitions,  in  accordance 
with  agency  procedures. 

On  EVM  for  managing  large  scientific,  engineering,  construction  and  environ¬ 
mental  cleanup  projects,  the  Department  of  Energy  (DoE) — like  other  govern¬ 
ment  agencies— sees  EVM  as  a  disciplined  methodology  that  integrates  the 
work  scope,  schedule  and  cost  parameters  of  high-value,  complex  projects  via 


Frank  is  the  director  of  the  Project  Management  Policy  and  Systems  ( PM-30 )  Division  in  the  De¬ 
partment  of  Energy  (DoE).  Kester  is  a  program  analyst  for  the  DoE's  PM-30  Division.  Urschel  is 
a  DoE  support  contractor  from  Humphreys  &  Associates ,  an  EVMS  consulting  company. 
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systematic  planning  and  effective  monitoring  and  control  to 
permit  a  detailed  assessment  of  overall  performance  during 
the  project's  life  cycle.  The  32  Electronic  Industries  Alliance 
(EIA)-748  Guidelines  outline  the  qualities  and  characteristics 
of  a  credible,  reliable  and  compliant  EVMS.  By  definition,  a 
compliant  EVMS  provides  for  the  generation  of  valid  and  veri¬ 
fiable  performance  data,  permits  the  evaluation  of  progress, 
and  allows  for  the  calculated  probability  of  meeting  program¬ 
matic  and  contractual  requirements  for  cost,  schedule  and 
technical  viability.  A  key  aspect  is  the  ability  to  capture  physi¬ 
cal  and  technical  progress  to  determine  what  "done"  looks 
like,  rather  than  what  work  has  been  done.  EVM  is  founded 
on  the  premise  that  project  teams  make  the  best  decisions 
when  they  have  the  best  information. 

While  EVMS  is  but  one  of  many  tools  used  to  plan  and  control 
project  performance,  decision  making  and  communications 


Guidelines  by  providing  greater  clarity  and  consistency  in  how 
the  department  does  its  business.  In  practice,  loosely  defined 
EVMS  Guidelines  have  led  to  long-term  inconsistencies  in  the 
interpretation  of  requirements.  The  release  of  the  DoE  EVMS 
Interpretation  Handbook  ( EVMSIH )  rectified  this  by  providing 
more  precise  definitions  and  translations. 

We  believe  this  to  be  a  positive  first  step  toward  addressing 
the  White  House  OMB's  Office  of  Federal  Procurement  Policy 
memorandum  titled  " Reducing  the  Burden  of  Certifying  Earned 
Value  Management  Systems. "  The  memorandum's  purpose  is 
to  encourage  agencies  to  enter  into  reciprocity  agreements 
to  recognize  EVMS  certification  across  agencies.  Reciprocity 
would  reduce  burdens  both  on  the  agencies  that  assess  EVMS 
capabilities  and  the  contractors  that  are  required  under  FAR 
Subpart  34.2  to  have  certified  EVMSs.  The  cost  of  obtain¬ 
ing  EVMS  certification  can  be  significant,  and,  under  current 


Big  data  are  at  the  core  of  this  different 
thinking— using  data  sets  and  algorithms 
to  summarize,  detect  patterns,  and  draw 
conclusions. 


are  impaired  when  EVMS  data  are  deficient  and  do  not  ac¬ 
curately  reflect  current  project  status  or  provide  acceptable 
forecasts.  This  means  that  the  project  performance  data 
including  cost  and  schedule  conditions,  and  the  progress 
made  toward  completing  contract  deliverables  by  the  proj¬ 
ect  manager  to  customers  and  stakeholders  become  highly 
subjective  rather  than  a  comprehensive  measurement  of  an 
unbiased  EVMS  to  highlight  inefficiencies  and  looming  risks 
to  schedule  and  budget. 

A  New  Strategy 

Paul  Bosco,  director  of  the  DoE's  Office  of  Project  Manage¬ 
ment  Oversight  and  Assessments,  is  challenged  with  dem¬ 
onstrating  to  departmental  leaders  a  way  of  implementing  an 
EVMS  strategy  that  consistently  provides  timely,  accurate  and 
reliable  project  information  in  the  most  cost-effective  manner. 
The  implementation  of  an  automated,  data-driven  analytics- 
based  EVMS  compliance  strategy  is  viewed  as  key  to  meeting 
the  DoE's  EVMS  strategic  goals  while  dramatically  improving 
the  assessment  of  a  contractor's  EVMS  compliance. 

For  years,  government  and  industry  have  debated  the  finer 
points  of  EVMS  compliance  with  no  real  result.  To  reach  com¬ 
mon  ground,  the  DoE  is  actively  working  to  ensure  that  in¬ 
dustry  understands  and  complies  with  the  32  EIA-748  EVMS 


practices,  contractors  sometimes  are  required  to  obtain  mul¬ 
tiple  agency  EVMS  certifications.  We  believe  that  providing 
one  face  of  government  to  industry  by  sharing  EVMS  review 
information  between  and  among  agencies,  with  each  following 
a  standard  automated  testing  protocol,  will  reduce  waste  and 
promote  efficiency. 

We  also  believe  that  an  automated  compliance  strategy  can 
eliminate  EVMS  compliance  barriers  that  result  in  some  gov¬ 
ernment  agencies  and  contractors  thriving  while  others  fail. 
Efficiency  through  automation  reduces  operating  costs,  is 
compatible  with  smaller  staffs  and  allows  agencies  to  oper¬ 
ate  with  fewer  resident  experts. 

It  has  often  been  observed  that  if  you  keep  doing  the  same 
things  you  generally  get  the  same  results.  By  extension,  if 
you  keep  doing  the  same  thing  in  an  environment  of  shrink¬ 
ing  budgets  and  changing  priorities,  you  probably  will  suffer 
worse  results. 

It  is  now  time  to  rethink  how  we  perform  EVMS  compliance  as¬ 
sessments  and  reviews  (both  initial  certification  and  follow-on 
surveillance).  Big  data  are  at  the  core  of  this  different  think¬ 
ing— using  data  sets  and  algorithms  to  summarize,  detect  pat¬ 
terns,  and  draw  conclusions.  A  data-driven  approach  is  used 
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Figure  1.  Typical  Process  Flow 


^Materiality  =  Root  Cause  Analysis  +  1  month 

Figures  and  table  by  the  U.S.  Department  of  Energy 


QUALIFYING  EXPECTATION-LINES  OF  INQUIRY  (QELOI)  TESTING  PROTOCOL 

5. A. 5.  Are  control  accounts  established  at  appropriate  levels  based  on  the  complex¬ 
ity  of  the  work  and  the  control  and  analysis  needed  to  manage  the  work  effectively? 

A  CAM  may  be  responsible  for  more  than  one  CA.  The  key  is  that  the  CAM  must  be  able 
to  demonstrate  effective  control  of  the  CA(s).  The  larger  the  staff,  the  more  CAs  open  at 
the  same  time  means  the  CAM,  all  other  factors  equal,  has  more  difficulty  demonstrating 
effective  control.  There  are  no  dollar/span  of  management  thresholds  limiting  a  CAM's 
responsibility.  A  CAM's  technical  background,  experience  and  time  devoted  to  the  CAM 
responsibilities  are  the  only  limits/factors  that  guide  how  many  and  the  scope  of  CA(s) 
one  CAM  can  be  responsible  for. 

COMPLIANCE  TESTING  PROTOCOL  BASIS 

1.  Identify  "Major"  control  accounts  (CAs)  if  the  following  are  met:  CA  >  7%  of  Total  BAC 
(and)  CA  >  10%  of  Period  BCWS 

2.  For  each  major  CA,  assess  span  of  control  by  monitoring  consecutive  CA  level  VAR 
trips  for  period  SV  or  CV  at  +/- 10%. 

3.  If  a  CA  trips  a  VAR  for  period  SV  or  CV,  for  (3)  consecutive  months,  a  "Red"  will  be 
recorded  for  the  project. 

4.  If  CA  trip  continues,  a  project  level  ribbon  chart  will  show  "red,"  and  will  continue  until 
the  CA  level  is  within  the  metric  threshold. 


Abbreviations  appearing  in  graphics:  ACWP  =  Actual  Cost  of  Work 
Performed;  BAC  =  Budget  at  Completion;  BCWP  =  Budgeted  Cost  of 
Work  Performed;  BCWS  =  Budgeted  Cost  of  Work  Scheduled;  CA 
=  Control  Account;  CAM  =  Control  Account  Manager;  CV  =  Cost 
Variance;  IMS  =  Integrated  Master  Schedule;  PMB  =  Performance 
Measurement  Baseline;  PP  =  Planning  Package;  SV  =  Schedule  Vari¬ 
ance;  VAR  =  Variance  Analysis  Report;  WP  =  Work  Package. 


to  efficiently  test  the  reliability  of  core  management  processes 
from  initial  implementation  and  continually  afterward  to  re¬ 
duce  the  risk  of  system  failure  during  sustainment  and  main¬ 
tenance.  The  capability  to  remotely  test  a  contractor's  EVMS 
data  greatly  reduces  and  may  eliminate  the  need  for  multiple 
government  assessments,  and  the  labor  and  travel  costs  as¬ 
sociated  with  numerous  visits  to  a  contractor's  plant. 

Piloting  an  Automated  Data-Driven  Protocol 

The  DoE  EVMS  compliance  process  uses  an  automated, 
data-driven  approach  to  efficiently  test  the  reliability  of  core 
management  processes  from  a  project's  initial  implementa¬ 
tion  and  continually  thereafter  to  reduce  the  risk  of  system 
failure.  Data-driven  testing  reveals  vital  information  about 
the  cause  of  a  deficiency  and  its  symptoms  (diagnosis)  and  is 
invaluable  in  monitoring  the  effects  of  any  corrective  actions. 
Automated  processing  shortens  the  assessment  cycle  with 
the  advancement  of  data  collection  and  analytical  techniques 
using  technology. 

The  DoE  EVMSIH  clarifies  the  intent  of  the  EIA-748  Guidelines 
through  the  use  of  Qualifying  Expectation— Lines  of  Inquiry 
(QELOI)  that  identify  the  qualities  and  characteristics  distin¬ 
guishing  each  guideline  from  the  others.  To  support  the  veri¬ 
fication  of  the  QELOI,  DoE  and  its  industry  partners,  both  di¬ 


rectly  and  via  the  Energy  Facility  Contractor's  Group  (EFCOG), 
developed  a  series  of  QELOI  Test  Steps  and  Test  Metrics  de¬ 
signed  to  measure  performance  of  each  of  these  attributes. 
These  Test  Steps  and  Test  Metrics  establish  EVMS  compliance 
parameters  for  evidenced-based  assessments. 

To  test  its  hypothesis  that  EVMS  compliance  can  be  accom¬ 
plished  using  an  automated,  data-driven  analytics-based  ap¬ 
proach,  the  DoE  entered  into  a  10-month  pilot  project  with  a 
contractor,  Consolidated  Nuclear  Security,  LLC,  in  Oak  Ridge, 
Tennessee,  to  determine  whether  observations  of  the  ap¬ 
proach  agree  with  or  conflict  with  the  predictions  derived  from 
a  hypothesis.  One  of  the  objectives  of  the  pilot  project  was  to 
produce  an  updated  EVMSIH  that  was  fully  tested,  appropri¬ 
ately  automated  (to  the  greatest  extent  practicable),  and  ready 
for  use  by  other  DoE  projects  subject  to  the  requirements  of 
DoE  Order  413. 3B,  "Program  and  Project  Management  for  the 
Acquisition  of  Capital  Assets.”  Those  requirements  provide 
program  and  project  management  direction,  inclusive  of  a 
compliant  EVMS,  for  acquiring  capital  assets  within  the  origi¬ 
nal  performance  baseline  scope,  cost  and  schedule. 

The  pilot  project  methodology  established  a  collaborative 
laboratory-type  environment  designed  to  develop  and  test 
a  comprehensive  EVMSIH  update,  while  understanding  the 
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Table  L  Test  Result  Example 


GUIDELINE 

TEST  NO. 

TEST  DESCRIPTION 

TEST  STEP 

THRESHOLD 

TRIP 

1 

1.A.1-1.1 

Verify  that  the  numeric  CA/WP/PP  WBS  codes  align  between  the  WBS  Dictionary  and  EVMS  Cost  Tool. 

01_A_01-01_01_d 

0.05 

NO 

1 

1.A.1-1.2 

Verify  that  the  numeric  CA/WP/PP  WBS  codes  align  between  the  IMS  Flat  File  and  the  EVMS  Cost  Tool. 

01_A_01-01_02_d 

0.05 

NO 

1 

1.A.2-1 .1 

Verify  the  Dollar  value,  at  the  CA/WP/PP  WBS  levels,  align  between  the  IMS  Flat  File  and  the  IPMR/CPR  Format  1. 

01_A_02-01_01_d 

0.05 

NO 

1 

1. A. 2-2.1 

Verify  the  WBS  Dictionary  and  Work  Authorization  scope  statements  align. 

01_A_02-02_01_d 

0.05 

YES 

2 

2.A.1-1.1 

Verify  the  dollar  values  at  the  functional  categories  align. 

02_A_01-01_01_d 

0 

NO 

2 

2.A.1-1.2 

Verify  the  total  PMB  dollar  value  is  the  same. 

02_A_01-01_02_d 

0 

NO 

2 

2.A.1-1.3 

Verify  the  RAM  to  the  Cost  Tool 

02_A_01-01_03_d 

0.05 

NO 

3 

3.  A. 1-1.1 

Verify  physical  percent  complete  aligns  between  IMS  and  EVMS  Cost  Tool  for  active  WPs. 

03_A_01-01_01_d 

0.05 

YES 

3 

3.A.1-2.1 

Verify  forecast  start  and/or  actual  start  dates  for  incomplete  CAs,  WPs  and  PPs  align  between  IMS  and  the  EVMS  Cost  Tool 
(Excluding  SVTs  and  Milestones). 

03_A_01-02_01_d 

0.05 

NO 

3 

3.A.1-3.1 

Verify  forecast  finish  and/or  actual  finish  dates  for  incomplete  CAs,  WPs  and  PPs  align  between  IMS  and  the  EVMS  Cost  Tool 
(Excluding  SVTs  and  Milestones). 

03_A_01-03_01_d 

0.05 

NO 

3 

3.A.1-4.1 

Verify  baseline  alignment  of  hours  between  the  WA  and  the  EVM  Cost  Tool  for  WP,  PP  and  CA. 

03_A_01-04_01_d 

0.05 

NO 

3 

3.A.1-4.2 

Verify  baseline  alignment  of  dates  between  the  WA  and  the  EVM  Cost  Tool  for  WP,  PP  and  CA. 

03_A_01-04_02_d 

0.05 

NO 

3 

3.A.1-4.3 

Verify  baseline  alignment  of  dollars  between  the  WA  and  the  EVM  Cost  Tool  for  WP,  PP  and  CA. 

03_A_01-04_03_d 

0.05 

NO 

3 

3.A.1-5.1 

Verify  the  charge  codes  align  at  the  CA/WP/PP  level  between  the  IMS  and  the  WA. 

03_A_01-05_01_d 

0.05 

NO 

3 

3.A.1-5.2 

Verify  baseline  dates  align  between  the  IMS  and  the  WA  (excluding  SVTs  and  Milestones)  for  CA/WP/PPs. 

03_A_01-05_02_d 

0.05 

NO 

3 

3.A.1-6.2 

Verify  dates  align  between  the  IMS  and  the  EVMS  Cost  Tool  for  CA/WP/  PPs. 

03_A_01-06_02_d 

0.05 

NO 

3 

3.A.1-6.3R1 

Verify  BAC  aligns  between  the  IMS  and  the  EVMS  Cost  Tool  for  incomplete  CA/WP/PPs. 

03_A_01-06_03R1_d 

0.05 

NO 

3 

3.A.1-6.4 

Verify  EOC  aligns  between  the  IMS  and  the  EVMS  Cost  Tool  for  incomplete  CA/WP/PPs. 

03_A_01-06_04_d 

0.05 

NO 

3 

3.A.1-6.5 

Verify  hours  align  between  the  IMS  and  the  EVMS  Cost  tool  for  incomplete  CA/WP/PP 

03_A_01-06_05_d 

0.05 

NO 

3 

3.  A. 1-7.1 

Verify  baseline  hours  and  total  dollars  at  CA  level  align  between  BCP  Log  and  WA. 

03_A_01-07_01_d 

0.05 

NO 

3 

3.A.1-8.1 

Verify  the  IMS  forecast  start  and  finish  dates  align  with  the  ETC  start  and  finish  dates  in  the  EVMS  Cost  Tool  for  active  CA/WP/PPs. 

03_A_01-08_01_d 

0.05 

NO 

3 

3.A.1-9.1 

Verify  the  RAM  dollar  values  align  to  the  CPR  Format  1  Dollar  Values. 

03_A_01-09_01_d 

0 

NO 

5 

5.A.1-1.1 

Verify  the  EVMS  Cost  Tool  has  only  one  OBS  assigned  to  each  CA. 

05_A_01-01_01_d 

0.05 

NO 

5 

5.A.1-2.1 

Verify  the  EVMS  Cost  Tool  has  only  one  WBS  assigned  to  each  CA. 

05_A_01-02_01_d 

0.05 

NO 

5 

5.A.1-3.1 

Verify  if  there  SLPPs  that  the  PMB-Total  CA  Budget-UB  =  SLPP  value. 

05_A_01-03_01_d 

0 

NO 

5 

5. A. 2-1.1 

Verify  there  is  a  CAM  assigned  to  each  CA  and  is  in  alignment  with  the  IMS  and  EVMS  Cost  Tool. 

05_A_02-01_01_d 

0.05 

NO 

5 

5. A. 3-1.1 

Verify  the  CAM  assignment  aligns  between  the  RAM  and  Work  Authorization. 

05_A_03-01_01_d 

0.05 

NO 

5 

5. A. 5-1.1 

To  verify  effective  CAM  Span  of  Control,  identify  the  Major  CAs  (>  7%  of  the  total  BAC  and  >10%  of  the  period  BCWS)  and  if  the 
current  period  SV%  and/or  CV%  greater  than  10% 

05_A_05-01_01_a 

0 

YES 

commercial  practices  used  in  managing  engineering  and 
construction  projects,  to  obtain  the  most  effective  and  ef¬ 
ficient  and  EIA-748-compliant  EVMS.  This  lab  environment 
employed  a  build-deploy-test  workflow  to  automate  cost- 
effective  EVMS  compliance  as  much  as  reasonably  achiev¬ 
able.  This  included  the  determination  of  thresholds  reflecting 
the  relative  importance  of  the  attributes.  There  would  then 
follow  an  iterative  process  to  collaboratively  evaluate  the  re¬ 
sults  and  refine  the  testing  specifications  to  confirm  expected 
outcomes.  Thus,  the  DoE  would  focus  only  on  those  test 
metric  values  that  are  outside  the  defined  threshold  value, 
getting  to  the  real  issues  and  reducing  EVMS  oversight  costs. 

The  expectation  is  that  by  documenting,  classifying  and  se¬ 
quencing  the  EVMSIH  QELOI,  contractors  will  be  able  to  find 
the  underlying  causes  of  core  management  process  issues  and 
self-correct  prior  to  any  government  involvement. 

Figure  1  shows  a  typical  process  flow  and  how  the  pilot  project 
went  about  defining,  interpreting,  and  documenting  EVMS 
compliance  requirements  for  each  EIA-748  Guideline.  Each 


process  flow  was  then  provided  to  the  developers  to  build 
the  code  necessary  to  extract  data  from  the  EVMS  and  the 
algorithm  to  calculate  the  result  and  assign  an  assessment 
result— green  (not  a  compliance  concern)  or  red  (a  compliance 
concern)— based  upon  a  predetermined  threshold. 

Table  1  shows  a  small  sample  of  the  test  results  from  the  au¬ 
tomated  calculation  identifying  the  test  results  from  the  auto¬ 
mated  calculation  identifying  those  QELOI  requiring  attention. 

The  contractor  responded  to  the  expressed  needs  of  the  pilot 
project,  providing  access  to  its  project  information  and  per¬ 
formance  data,  and  dedicating  key  resources  throughout  the 
10-month  effort.  The  contractor  also  responded  by  making  a 
significant  commitment  and  investment  in  development  and 
deployment  of  new  testing  technology,  thus  accelerating  the 
automation  of  EVMS  testing.  The  robust  collaboration  and 
collegiality  validated  the  objectives  of  the  pilot  project,  which 
involved  alignment  of  EVMS  implementation  interpretations 
and  the  reduction  of  costs,  while  preserving  individual  public 
and  private  identities  and  missions.  Several  key  lessons  can 
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be  learned  from  the  pilot  project  experience  and  applied  to 
similar  environments: 

■  EVMS  assessments  can  come  through  both  automated 
and  manual  testing. 

■  The  method  chosen  comes  down  to  the  ability  to  digitize 
the  analog  process. 

■  More  assessments  can  be  accomplished  and  issues  iden¬ 
tified  quicker  using  an  automated  method. 

■  The  team  is  able  to  react  more  quickly,  saving  time  and 
money. 

■  Innovation  requires  a  mindset  shift  toward  trusting  the 
data. 

It  is  important  to  stay  the  course  and  pursue  the  goal  regard¬ 
less  of  obstacles  or  criticism. 

Eliminating  Excess 

At  the  start  of  the  pilot  project,  the  number  of  EVMSIH  QELOI, 
Test  Steps,  and  Test  Metrics  were  134,  345  and  597,  respec¬ 
tively.  Of  the  latter,  481  Test  Metrics  (81  percent  of  the  total) 
were  initially  categorized  as  requiring  a  manual  approach  to 
determine  El  A-748  compliance.  The  Test  Metric  to  QELOI  ratio 
was  approximately  4.5  to  1.  Based  on  expert  opinion,  the  op¬ 
timum  ratio  is  on  the  order  of  2  or  3  Test  Metrics  to  1  QELOI, 
while  still  providing  a  valid  basis  for  testing  EVMS  compliance. 
The  challenge  faced  by  the  Pilot  Project  was  picking  the  right 
number  and  level  of  Test  Metrics  and  the  quality  of  the  data 
gathered  to  ensure  the  effectiveness  of  the  analysis  and  inter¬ 
pretation  of  the  automated,  data-driven  EVMS  compliance. 
Figure  2  displays  the  Pilot  Project  final  totals. 

A  live  demonstration  of  DoE's  automated,  data-driven  ap¬ 
proach  was  provided  at  the  January-February  2017  National 
Defense  Industrial  Association  (NDIA)  working  meeting  in 
Orlando,  Florida.  The  proof-of-concept  demonstration  pro¬ 
cessed  hundreds  of  thousands 
of  contractor  EVMS  data  ele¬ 
ments  for  236  Test  Metrics.  It 
took  mere  minutes,  using  a  flat 
file  to  access  data  base  archi¬ 
tecture,  to  verify  the  validity 
of  automating  a  manual  task 
using  hard  copy  artifacts  that 
previously  required  days  if  not 
weeks  and  months  to  assemble, 
calculate,  observe,  conclude 
and  document.  We  believe  the 
benefits  (i.e.,  clarity,  consis¬ 
tency  and  cost)  associated  with 
automated,  data-driven  EVMS 
compliance  can  better  position 
the  DoE  and  other  government 
agencies  to  meet  all  govern¬ 
ment  requirements  at  best 
value  while  providing  an  optimal 
project  controls  tool. 


The  next  step  in  exporting  the  automated,  data-driven  EVMS 
compliance  process  across  the  DoE  complex  is  to  replicate 
and  tailor  where  necessary  the  results  of  the  pilot  project  with 
Los  Alamos  National  Security,  LLC,  at  the  Los  Alamos  National 
Laboratory  in  Los  Alamos,  New  Mexico. 

Transforming  Into  the  Digital  Realm 

When  looking  to  an  EVMS  future,  one  of  the  most  important 
questions  is  what  technology  can  be  added  to  an  organiza¬ 
tion's  operations.  We  believe  that  the  challenges  facing  project 
managers  and  their  teams  to  deliver  accurate  and  reliable  data, 
and  to  provide  adequate  checks  and  balances,  can  only  be  met 
by  looking  at  new  ways  of  doing  things.  Technology  must  play 
a  key  role  in  delivering  this. 

In  the  future,  everyone  conducting  EVMS  compliance  will 
need  extensive  knowledge  of  the  technology  solution  to  exe¬ 
cute  the  automated,  data-driven  methodology.  To  get  the  ac¬ 
tionable  information  they  need  from  the  EVMS,  both  govern¬ 
ment  and  contractor  personnel  alike  will  perform  diagnostic 
tests  with  the  data  to  determine  the  presence,  materiality  and 
systemic  nature  of  the  issue.  Test  results  will  be  evaluated 
based  on  predetermined  thresholds  to  measure  compliance 
and  indicate  when  corrective  action  may  be  needed. 

In  the  end  state,  a  digital  dashboard  will  let  customers  and 
stakeholders  know  the  compliance  (or  health)  status  of  the 
EVMS  performance  data  upon  which  they  are  making  financial 
and  other  programmatic  decisions. 

Any  measurement  that  falls  outside  historic  norms  or  devi¬ 
ates  from  a  predetermined  threshold  will  be  flagged.  Within 
minutes,  government  will  know  what  issues  to  focus  on  and 
whether  additional  engagement  with  a  contractor  is  required. 
The  future  direction  for  EVMS  compliance  efforts  will  be  based 
on  these  results.  DoE's  level  of  engagement  with  a  contractor 


Figure  2.  Final  Test  Metric  Statistics 

Total  Test  Metrics  by  Process  Area 


Automated  vs. 
Manual  Tests 
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will  be  carried  out  through  a  three-stage  process,  beginning 
with  a  less  formal  approach.  This  should  engender  a  more 
collaborative  approach  and  produce  quicker  results. 

The  stages  are: 

■  Stage  1:  Limited  Engagement.  When  the  tests  do  not  exceed 
thresholds,  the  results  will  be  shared  with  the  contractor, 
and  DoE  will  minimize  additional  surveillance  activities. 

■  Stage  2:  Investigation.  When  any  test  exceeds  the  thresh¬ 
olds,  DoE  will  discuss  the  issue  with  the  contractor,  which 
will  investigate  whether  a  breach  has  occurred.  Based  on 
the  investigation,  DoE  may  issue  advice  to  clarify  and/or 


way  to  emerging  issues,  allowing  a  contractor  to  take  preemp¬ 
tive  action  to  remedy  data  quality  issues  before  they  escalate 
to  major  deficiencies. 

System  Maturity  Scoring 

The  DoE  approach  is  data  rich,  taking  the  EVMS  compliance 
assessment  down  to  the  core  attributes  of  each  EIA-748 
Guideline ,  through  the  testing  of  management  processes,  in¬ 
cluding  organizing,  work  authorization,  planning,  scheduling 
and  budgeting,  forecasting  and  change  control.  Next,  value 
weightings  can  be  assigned  to  each  QELOI  for  appraising  the 
maturity  of  an  EVMS.  Similar  to  how  FICO  Scores  are  cal¬ 
culated  from  many  different  pieces  of  credit  data  in  a  credit 


With  fewer  EVMS  personnel  needed, 
DoE  will  depend  heavily  on  technology  to 
remotely  test  a  contractor's  EVMS  data. 


interpret  the  relevant  provisions  of  the  attribute  interpreta¬ 
tion  for  the  contractor  to  follow. 

■  Stage  3:  Corrective  Action.  If  the  issue  continues,  DoE  will 
require  a  more  formal  investigation  by  the  contractor  that 
will  include  a  root-cause  analysis  and  corrective  action  plan. 
Throughout  this  process,  government  personnel  will  ensure 
that  concerns  are  promptly  considered  by  the  contractor  and 
that  any  necessary  action  is  taken  as  swiftly  as  possible. 

Remote  Diagnostics 

Automation  and  mobility  are  central  to  DoE  EVMS  compli¬ 
ance.  One  of  its  core  functions  is  designing,  developing  and 
deploying  a  Web-based  content  delivery  system.  Within 
the  DoE,  that  is  the  Project  Assessment  and  Reporting  Sys¬ 
tem  (PARS),  the  flagship  enterprise  system  of  record  for 
project  data  and  information.  The  more  automation  and 
mobile  capabilities  PARS  can  provide  to  the  EVMS  compli¬ 
ance  process,  the  more  it  will  help  move  the  mission  toward 
greater  consistency  and  clarity  and  consolidate  operations 
for  cost  savings. 

A  small  and  dedicated  group  of  DoE  subject-matter  experts 
control  PARS  capability  development,  configuration  manage¬ 
ment  and  systematic  functionality  upgrades  throughout  the 
system's  life  cycle.  With  fewer  EVMS  personnel  needed,  DoE 
will  depend  heavily  on  technology  to  remotely  test  a  contrac¬ 
tor's  EVMS  data.  This  will  eliminate  the  need  for  multiple  DoE 
assessments  and  costs  and  time  of  numerous  visits  to  a  con¬ 
tractor's  facility.  PARS  will  mark  suspect  data  and  point  the 


report,  an  EVMS  maturity  score  can  reflect  how  compliant  or 
noncompliant  a  contractor's  EVMS  is  at  any  point  in  the  proj¬ 
ect  life  cycle.  QELOI  weighting  could  be  adjusted  accordingly 
to  best  implement  the  EVMS  through  the  different  phases 
of  the  project. 

Conclusions 

The  first  step  is  realizing  the  importance  of  EVMS  as  a  key 
project  management  tool— and  compliance  is  a  government 
requirement.  Agencies  must  reinvigorate  the  government 
benefits  from  its  implementation,  while  being  prudent  and 
financially  responsible.  We  must  work  to  achieve  maximum 
value  of  this  tool  in  the  most  cost-effective  manner. 

This  article  has  presented  several  insights  into  how  DoE  has 
changed  its  approach  to  EVMS  compliance.  There  is  a  needed 
and  inevitable  journey  away  from  single  points  of  expertise 
and  gut  feel  toward  a  smarter  EVMS  compliance,  aided  by  the 
power  of  data,  analytics  and  technology. 

However,  all  efforts  to  rethink  EVMS  compliance  are  meaning¬ 
less  unless  the  EVM  methodology  and  the  associated  EVMS 
are  simplified  to  be  timely  and  intuitive  and  allow  both  gov¬ 
ernment  and  contractor  project  managers  and  their  teams  to 
realize  the  greatest  benefit.  When  this  happens,  EVMS  compli¬ 
ance  becomes  a  management  standard  operating  procedure 
and  not  a  burdensome  requirement. 


The  authors  can  be  contacted  at  melvin.frank@hq.doe.gov;  david.kester@ 
hq.doe.gov;  and  karen.urschel@hq.doe.gov. 
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ince  2005,  the  U.S.  Government 
Accountability  Office  (GAO)  has 
developed  and  distributed  a  series 
of  guides  that  address  shortcom¬ 
ings,  recommend  best  practices, 
as  well  as  related  case  studies  regarding 
integrated  program  management  (IPM). 
All  were  developed  to  support  improved 
performance  and  end  results  of  federal 
government  programs  through  better 
planning,  risk  awareness  and  enhanced 
management  decision  making. 


Overview 

Best  practices  guides  are  compilations  that  federal  organizations  and 
industry  use  to  develop  and  maintain  reliable  project  controls  and 
management  tools  throughout  the  life  of  a  government  acquisition 
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Got  opinions  to  air? 

Interested  in  passing  on  lessons  learned  from 
your  project  or  program? 

Willing  to  share  your  expertise  with  the 
acquisition  community? 

Want  to  help  change  the  way  DoD  does 
business? 

Write  an  article  (1,500  to  2,500  words)  and  Defense  AT&L 
will  consider  it  for  publication.  Our  readers  are  interested  in 
real-life,  hands-on  experiences  that  will  help  them  expand 
their  knowledge  and  do  their  jobs  better. 

What’s  in  It  for  You? 

First  off,  seeing  your  name  in  print  is  quite  a  kick.  But  more 
than  that,  publishing  in  Defense  AT&L  can  help  advance  your 
career.  One  of  our  authors  has  even  been  offered  jobs  on 
the  basis  of  articles  written  for  the  magazine. 

Now  we  can’t  promise  you  a  new  job,  but  many  of  our 
authors: 

•  Earn  continuous  learning  points 

•  Gain  recognition  as  subject-matter  experts 

•  Are  invited  to  speak  at  conferences  or  symposia 

•  Get  promoted  or  rewarded 

For  more  information  and  advice  on  how  to  submit  your 
manuscript,  check  the  writer’s  guidelines  at  https://www. 
dau.mil/library/defense-atl/p/Writers-Guidelines  or  contact 
the  managing  editor  at  datl@dau.mil. 


program.  Guides  can  be  used  across  the  federal  government 
for  developing,  managing  and  evaluating  various  aspects  of 
federal  acquisition  programs.  Guides  provide  principles  for 
GAO  auditors  to  use  in  assessing  various  aspects  of  acquisi¬ 
tion  programs. 

It  is  important  to  note  that  all  guides  were  developed  in  col¬ 
laboration  with  experts  in  one  or  all  of  the  cost,  schedule 
or  technical  management  disciplines  covered  by  the  best 
practice  guides.  This  approach  was  implemented  with  the 
development  of  the  first  best  practice  guide  and  has  become 
a  well-understood  process  that  has  been  applied  to  other 
publications.  The  process  itself  involves  GAO  seeking  input 
from  a  wide  range  of  experts,  starting  with  planning  and  de¬ 
sign  through  report  development. 

GAO's  best  practice  guides  are  developed  through  an  iterative, 
consultative  process  that  involves  a  committee  of  experts  in 
the  related  domain(s).  Members  are  from  government  agen¬ 
cies,  private  companies,  independent  consultant  groups,  trade 
industry  groups  and  academia.  On  the  Cost  Guide ,  comments 
were  invited  from  nearly  1,000  experts  representing  a  broad 
range  of  industries,  international  perspectives  and  program 
environments.  Expert  meetings  are  open  to  anyone  with  the 
requisite  experience  and  interest  in  the  topic.  Meeting  minutes 
are  extensively  documented  and  archived  after  review  by  the 
GAO  team  and  all  participants. 

The  foundational  guides  that  have  been  published  to  date  and 
are  in  development  are  as  follows: 

The  Cost  Estimating  and  Assessment  Guide 
( GAO-09-3SP ) 

GAO's  original  intention  was  that  the  Guide  would  provide 
GAO  auditors  with  a  standardized  approach  for  analyzing  pro¬ 
gram  costs.  Internal  GAO  research  found  that  federal  guide¬ 
lines  were  limited  on  processes,  procedures  and  practices  for 
ensuring  credible  cost  estimates.  The  team  concluded  that 
GAO  could  best  serve  the  Congress  and  the  federal  commu¬ 
nity  by  filling  that  gap.  GAO  began  drafting  the  Guide  in  2005, 
issued  a  draft  release  in  2007  for  public  comment  and  officially 
released  the  Guide  in  2009. 

The  purpose  of  the  Cost  Assessment  Guide  is  to: 

■  Address  generally  accepted  best  practices  for  ensuring 
credible  program  cost  estimates  applicable  across  gov¬ 
ernment  and  industry. 

■  Provide  an  explicit  link  between  cost  estimating  and 
earned  value  management  (EVM). 

■  Discuss  how  both  cost  estimating  and  EVM  are  needed 
for  setting  realistic  program  baselines  and  managing  risk. 

Schedule  Assessment  Guide  (GAO-16-89G) 

The  Schedule  Assessment  Guide  expands  on  the  scheduling 
concepts  introduced  in  the  draft  Cost  Guide  and  was  intended 
to  be  an  appendix  to  the  official  release  of  the  Cost  Guide  but 
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instead  was  issued  as  a  publication  in  its  own  right.  Drafting 
of  the  Guide  began  in  2010;  a  draft  was  issued  for  an  18-month 
comment  period  in  2012,  and  the  final  draft  was  issued  in  2015. 

The  purpose  of  the  Schedule  Assessment  Guide  is  to  allow  GAO 
auditors  to  assess  the  reliability  of  reported  dates  through  an 
assessment  of  project  schedules.  It  is  also  a  useful  resource 
for  agencies  to  create  or  append  existing  policies  and  guidance 
on  creating  and  maintaining  project  schedules. 


Agile  Implementation  Guide  (Draft) 

This  Guide  will  fill  a  criteria  void  on  "cost,"  "schedule"  and 
"performance"  (particularly  technical)  in  the  "cost-schedule- 
performance"  trio  of  management  elements  of  capital  acquisi¬ 
tion  programs  when  specifically  auditing  programs  utilizing  an 
Agile,  rather  than  traditional,  software  development  approach. 
It  also  addresses  the  cultural  and  organizational  changes  that 
are  needed  for  Agile  development  to  effectively  work  in  a  gov¬ 
ernment  agency.  Drafting  of  the  Guide  began  in  2016;  a  draft 


On  the  Cost  Guide ,  comments  were  invited 
from  nearly  1,000  experts  representing 
a  broad  range  of  industries,  international 
perspectives  and  program  environments. 


Technology  Readiness  Assessment  Guide 
(GAO-16-410G) 

The  Technology  Readiness  Assessment  Guide  fills  a  criteria  void 
on  "performance"  (particularly  technical)  in  the  "cost-sched- 
ule-performance"  trio  of  management  elements  of  capital 
acquisition  programs.  Drafting  of  the  Guide  began  in  2013,  a 
draft  was  released  for  a  12-month  comment  period  in  2015, 
and  the  final  is  scheduled  to  be  issued  in  2018. 

This  Guide  was  designed  to  bring  understanding  and  practice 
of  technology  readiness  assessments,  invented  decades  ago 
by  NASA  and  utilized  extensively  in  the  Department  of  De¬ 
fense,  but  now  available  to  other  agencies  that  lack  a  large 
technical  staff.  It  also  allows  GAO  auditors  to  assess  the  reli¬ 
ability  of  the  identification  and  management  of  technologies 
critical  to  the  success  of  a  given  capital  acquisition  program.  It 
is  intended  to  help  agencies  create  or  append  existing  policies 
and  guidance  on  creating  and  maintaining  technology  readi¬ 
ness  assessments,  whether  they  are  for  ongoing,  day-to-day 
project  management  or  major  milestone  decision  points. 

Cost  Estimating  and  Assessment  Guide — 

Update  Document 

This  revised  version  of  the  Cost  Guide  updates  the  original 
2009  Cost  Guide  and  is  intended  to  improve  definitions  of 
criteria  and  leading  practices  based  on  lessons  learned  over 
8  years  of  application  and  to  provide  the  latest  industry  and 
government  practices  along  with  updated  references  to  exist¬ 
ing  laws  and  federal  guidance.  This  Guide  provides  updated 
graphics,  more  recent  case  studies,  and  provides  integration 
with  the  Schedule  Guide ,  the  Technology  Readiness  Guide ,  the 
Agile  Guide ,  and  Standards  for  Internal  Control.  Drafting  began 
in  2016,  with  release  scheduled  for  2018. 


is  expected  to  be  released  for  a  24-month  comment  period  in 
2019,  with  final  issuance  probably  in  2021. 

The  Agile  Guide  is  intended  to  foster  a  better  understanding 
and  practice  of  Agile  software  development,  codified  in  a  2001 
manifesto  drafted  by  software  development  experts  in  the  pri¬ 
vate  sector,  but  now  utilized  extensively  by  federal  agencies 
and  soon  to  be  promoted  as  the  preferred  approach  to  soft¬ 
ware  development  by  the  Office  of  Management  and  Budget. 
It  is  intended  to  allow  GAO  auditors  to  assess  the  reliability  of 
the  management  of  processes  and  technologies  critical  to  the 
success  of  capital  acquisition  programs  using  an  incremental 
or  Agile  approach  to  software  development  and  the  risk  as¬ 
sumed  by  federal  agencies  in  transitioning  from  a  traditional 
to  Agile  approach  to  software  development. 

The  Agile  Guide  is  meant  to  be  useful  for  agencies  seeking  to 
create  or  append  existing  policies  and  guidance  on  develop¬ 
ing  software  using  an  Agile  approach,  whether  for  ongoing, 
day-to-day  project  management  or  overarching  organiza¬ 
tional  policy. 

All  GAO  best  practice  guides  are  available  free  of  charge  online 
by  browsing  at  http://www.gao.gov/search. 

All  of  the  best  practice  guides  include  numerous  case  study 
examples  highlighting  the  advantages  and  disadvantages  of 
varying  degrees  of  following  their  recommended  practices. 
These  case  studies  are  also  documented  and  available  at  the 
same  Web  resources  free  of  charge  for  those  wishing  to  ex¬ 
plore  the  topics  in  greater  depth.  & 


The  author  can  be  contacted  at  richeyk@gao.gov. 
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EARNED  VALUE  MANAGEMENT 


Jerald  Kerby  ■  Matoka  Forbes  ■  Stefanie  Terrell 


Sixth  in  a  series  of  articles 

The  Civilian  Agencies  and  Industry  Working  Group  (CAIWG)  on  Earned  Value  Man¬ 
agement  (EVM)  recently  released  its  guidance  on  Earned  Value  Management  System 
(EVMS)  reciprocity  and  EVM  scalability.  Both  initiatives  resulted  in  products  that  can 
be  used  to  help  lower  the  burden  and  cost  of  implementing  EVM.  The  initiatives  can  be 
found  at  http://caiwg.mycpm.org. 

In  today's  competitive  environment,  it  is  imperative  that  both  government  and  industry  seek  ways  to  reduce  costs 
and  gain  efficiencies.  There  is  continuing  discussion  about  the  cost  of  implementing  EVM,  a  disciplined  process 


Kerby  is  NASA's  Earned  Value  Management  (EVM)  Program  Executive  and  chairman  of  the  Working  Group  that  supports  EVM  implemen¬ 
tation  and  development  of  NASA's  EVM  policy  and  handbooks.  He  also  is  responsible  for  providing  both  classroom  and  hands-on  EVM  and 
data  analysis  training  to  the  project  management  community  at  NASA  and  the  Marshall  Space  Flight  Center  in  Alabama.  He  has  a  bachelor 
of  science  degree  from  Alabama's  Athens  State  University  and  a  master  of  science  degree  from  the  Florida  Institute  of  Technology.  Forbes 
is  the  Federal  Aviation  Administration  (FA A)  EVM  Focal  Point.  She  is  an  active  member  of  the  Civilian  Agency  Industry  Working  Group  and 
ensures  the  EVM  implementation  on  FA  A  programs.  Terrell  is  a  program  analyst  with  more  than  8  years  of  industry  and  government  experi¬ 
ence  on  all  matters  related  to  scheduling  and  EVM. 
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Though  not  a  formal  member,  the  Department  of  Defense 
(DoD)  participates  in  this  group  to  help  ensure  consistency 
in  the  ^hicition  of  project  management  principles  andlVM 

it^  across  the  governmeniind  industr 


that  identifies  cost  and  schedule  variances  primarily  based  on 
comparisons  between  planned  work  and  performed  work.  To 
date,  numerous  studies  have  offered  insight  into  the  real  costs 
of  using  EVM.  An  article  in  the  January-February  2017  issue 
of  Defense  AT&L  ("EVM  System's  [EVMS]  High  Cost— Fact 
or  Fiction?")  concluded  that  EVM  can  be  done  in  an  efficient 
manner  at  a  low  cost  if  implemented  correctly. 

The  C  Al  WG  was  created  to  bring  civilian  agencies  together  to 
share  best  practices  for  implementing  EVM  and  to  promote 
consistent  integrated  project  management  across  agencies. 
Though  not  a  formal  member,  the  Department  of  Defense 
(DoD)  participates  in  this  group  to  help  ensure  consistency 
in  the  application  of  project  management  principles  and  EVM 
requirements  across  the  government  and  industry. 

Reciprocity  Agreement 

The  intent  of  EVMS  reciprocity  is  to  create  an  established, 
universally  applicable  set  of  standards  for  use  in  conducting 
reviews  across  the  government.  The  desire  is  to  find  a  standard 
generic  enough  to  address  the  requirements  and  objectives  of 
any  agency,  yet  specific  enough  to  ensure  that  uniform  rules 
are  applied.  This  approach  provides  added  value  when  EVM 
review  reports  are  shared  between  agencies  and  departments 
and  used  to  reduce  the  need  for  duplicative  reviews. 

The  Office  of  Management  and  Budget  (OMB)  supported  this 
reciprocity  initiative.  A  presidential  memo  was  issued  on  Oct. 
23, 2015,  to  address  reducing  the  burden  of  certifying  Earned 
Value  Management  Systems.  It  stated: 

...  agencies  are  encouraged  to  enter  into  reciprocal  agreements 
with  other  agencies  and  to  post  their  EVM  processes  and  pro¬ 
cedures  on  their  public  websites.  Sharing  information  should 
help  identify  redundancy  in  the  certification  processes.  If  an 
agency  determines  that  its  certification  process  is  substan¬ 
tially  similar  to  the  certification  process  of  another  agency,  the 
agency  should  consider  whether  it  is  feasible  to  enter  into  an 
agreement  with  the  other  agency  for  the  mutual  recognition 
of  the  EVMS  certification. 

On  this  topic,  the  OMB  released  a  statement  in  2016  by  Anne 
Rung,  former  administrator  for  the  Office  of  Federal  Procure¬ 
ment  Policy,  which  stated: 


We're  committed  to  reducingthe  burden  on  both  government 
agencies  and  contractors,  and  our  October  2015  guidance  en¬ 
couraging  the  use  of  these  reciprocity  agreements  for  EVM 
system  certifications  is  a  significant  step  in  that  direction.  If 
agencies  recognize  each  other's  certifications,  we  can  avoid 
costly,  duplicative  compliance  reviews.  NASA  [National  Aero¬ 
nautics  and  Space  Administration]  and  the  FAA  [Federal  Avia¬ 
tion  Administration]  are  leaders  in  this  area— having  been  the 
first  to  sign  an  agreement  recognizing  each  other's  EVM  sys¬ 
tem  certifications  earlier  [in  2016]— and  we  encourage  other 
agencies  to  consider  a  similar  arrangement. 

This  memorandum  of  understanding  (MOU)  is  intended  to 
serve  as  an  umbrella  agreement  that  provides  for  the  shar¬ 
ing  of  EVMS  compliance  reports  between  federal  agencies. 
Agencies  retain  the  right  to  make  their  own  determination  of 
system  compliance  in  accordance  with  each  organization's 
policies  and  procedures.  It  is  also  anticipated  that  agencies 
would  enter  into  supplemental  agreements  providing  more 
specific  details  tailored  to  the  unique  requirements  of  future 
projects  and  partnerships. 

The  first  EVM  reciprocity  MOU  was  established  by  NASA  and 
the  FAA  in  March  2016.  That  MOU  states  that  recognition 
of  another  agency's  certification,  as  appropriate,  eliminates 
duplicative  compliance  reviews  and  results  in  cost  savings  for 
both  the  contractor  and  the  taxpayer.  Both  agencies  agree  that 
the  other  participant  will  recognize  the  supplier's  EVMS  as 
EIA-748  (EVMS  Standard)  compliant  and  will  ensure  that  the 
system  is  continuously  maintained.  This  MOU  could  result  in 
significant  savings  to  the  government  and  contractors,  given 
that  an  EVMS  review  can  cost  $1  million  to  $2  million  and  can 
take  a  few  years  to  complete. 

In  December  2016,  DoD  also  issued  an  umbrella  EVMS  reci¬ 
procity  MOU,  which  established  the  sharing  of  EVMS  review 
information  with  the  aim  of  increasing  the  efficiency  with 
which  agencies  determine  a  supplier's  EIA-748  compliance. 
The  MOU  further  recognizes  that  collaboration  between  agen¬ 
cies  will  help  improve  that  efficiency.  Under  this  reciprocity 
MOU,  the  FAA  currently  accepts  EVMS  certifications  for  in¬ 
dustry  by  the  Administrative  Contracting  Officer  of  the  DoD's 
Defense  Contract  Management  Agency.  In  addition,  NASA 
has  a  standing  MOU  with  DCMA  to  delegate  EVMS  reviews 
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Figure  L  Scaling  the  EVM  to  Project 
Needs 


of  the  project  increases,  so  does  the  project  controls  require¬ 
ment,  as  indicated  in  Figure  1  (which  initially  appeared  in  the 
NDIA's  Earned  Value  Management  System  Guideline  Scalability 
Guide ,  issued  in  November  2016). 

Figure  2,  also  drawn  from  the  NDIA  Scalability  Guide ,  is  an  ex¬ 
ample  of  a  responsibility  assignment  matrix  for  a  small  proj¬ 
ect  with  a  limited  number  of  control  accounts.  As  the  project 
grows  in  size  and  complexity,  the  application  of  earned  value 
management  techniques  also  will  increase. 

The  Scalability  Guide  also  is  intended  for  suppliers  that  may  not 
have  a  contractual  requirement  in  place  for  EVM  but  would 
like  to  benefit  from  implementing  EVM  practices.  The  Guide  is 
intended  for  industry  and  government  personnel  within: 


Source  of  graphics:  The  National  Defense  Industry  Association's  Earned 
Value  Management  System  Guideline  Scalability  Guide. 

and  surveillance  activities  among  companies  for  which  DoD 
already  does  this  type  of  work. 

The  key  to  progress  in  EVM  reciprocity  is  for  those  government 
agencies  that  are  heavily  involved  with  EVM— including  DoD, 
NASA,  FAA,  the  National  Reconnaissance  Office  (NRO)  and 
the  Department  of  Energy— to  establish  a  government-level 
standard  for  recognizing  an  EVMS.  In  the  future,  OMB  and  the 
CAIWG  expect  to  see  multiple  agencies  implement  EVM  reci¬ 
procity  agreements  and,  over  the  long  term,  employ  a  single, 
government-wide  reciprocity  agreement. 


■  Entities  such  as  universities,  laboratories,  small  busi¬ 
nesses  and  suppliers  with  small  to  midsize  projects 

■  Large  organizations  with  small  projects  that  issue  con¬ 
tracts  to  small  businesses  or  other  suppliers  that  may  not 
have  an  existing  EVMS  in  place 

■  Any  government  agency  with  small  contracts  and  an  EVM 
requirement 

Although  this  document  is  primarily  targeted  at  contractors, 
there  is  a  requirement  within  government  agencies  and  orga¬ 
nizations  to  follow  the  process  outlined  in  the  OMB  Circular 
A-11,  Capital  Planning  Guide  for  government  in-house  projects 
with  development  work.  The  EVM  Scalability  Guide  can  provide 
government  project  teams  with  information  on  optimizing  their 
EVM  implementation  based  on  each  project's  characteristics. 


Scalability  Guide  The  Scalability  Guide  is  intended  to  provide  guidance  for  set- 

in  addition,  the  CAIWG  developed  an  EVMS  Scalability  Guide  ting  up  and  establishing  fundamental  project  management 
that  outlines  how  EVM  imple¬ 


mentation  can  be  scaled  to  meet 
varying  project  needs.  The  Guide 
was  developed  specifically  for 
organizations  that  typically  don't 
have  contracts  that  require  EVM, 
or  those  lacking  the  large  infra¬ 
structure  typically  supporting  a 
fully  compliant  EVMS.  This  Guide 
was  jointly  developed  with  indus¬ 
try  and  is  now  part  of  the  National 
Defense  Industry  Association 
(NDIA)  document  library. 

Historically,  EVM  has  been  used 
on  large  development  and  pro¬ 
duction  contracts,  so  prior  to  the 
EVMS  Scalability  Guide  little  guid¬ 
ance  existed  for  smaller  projects. 
The  Guide  enables  EVM  scalabil¬ 
ity  based  on  project  size,  cost, 
risks,  complexity  and  other  fac¬ 
tors.  As  the  size  and  complexity 


Figure  2.  Responsibility  Assignment  Matrix 
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processes  rather  than  focusing  on  EVM-specific  information. 
The  Guide's  initial  chapters  cover  planning  and  organizing 
the  work,  such  as  establishing  a  Work  Breakdown  Structure 
(WBS)  and  assigning  responsibility.  Subsequent  topics  include 
scheduling  work  and  establishing  budgets.  The  Guide  then 
transitions  into  the  execution  phase  of  the  project:  managing 
a  baseline,  controlling  changes,  managing  subcontracts  and 
material,  controlling  costs  and  analyzing  data  that  can  be  used 
to  make  project  management  decisions. 

The  Scalability  Guide  has  received  a  positive  response  both 
from  industry  and  government  partners,  largely  because  the 
Guide  effectively  takes  into  consideration  all  32  guidelines  from 
the  EIA-748  EVMS  Standard.  It  further  provides  tips  and  sug¬ 
gestions  for  each  guideline  and  identifies  ways  they  can  be 
scaled,  thus  gaining  the  full  value  of  EVM  while  reducing  the 
burden  of  implementing  a  compliant  EVMS. 

Small  organizations  without  the  full  benefit  of  a  corporate 
EVMS  also  should  realize  a  significant  benefit  from  the  Guide. 
For  example,  a  university's  accounting  system  may  be  limited 


to  collecting  labor  costs  at  the  top  WBS  level  of  a  project.  The 
guide  offers  options  for  establishing  budgets  and  collecting 
costs  when  an  organization's  accounting  system  is  under  con¬ 
straints.  It  also  discusses  workaround  plans  to  address  other 
limitations  within  other  business  systems  that  support  EVM. 

In  summary,  discussion  of  the  cost  and  burden  of  EVM  is 
not  likely  to  decrease  any  time  soon.  The  civilian  agencies 
have  taken  the  initiative,  however,  to  reduce  the  burden 
and  costs  associated  with  EVM  by  developing  EVMS  reci¬ 
procity  agreements  and  the  EVM  Scalability  Guide ,  which 
respectively  can  reduce  duplicative  future  EVMS  reviews 
and  provide  guidance  for  small  projects  to  customize  EVM 
for  optimal  project  management. 

For  more  information  on  the  CAIWG  activities,  visit  the  web¬ 
site  at  http://caiwg.mycpm.org.  & 


The  authors  can  be  contacted  atjerald.g.kerby@nasa.gov;  matoka. 
forbes@faa.gov  and  stefanie.m.terrell@nasa.gov. 


Where  Can  You  Get  the  Latest  on  the 

Better  Buying  Power 

Initiatives? 


■  BBP  Gateway  (https://acc.dau.mil/bbp)  is  your  source  for  the 
latest  information,  guidance  and  directives  on  Better  Buying 
Power  in  defense  acquisition 

■  BBP  Public  Site  (https://acc.dau.mil/bbp)  is  your  forum 
to  share  BBP  knowledge  and  experience 
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Improving  Threat  Support 

for  DoD  Acquisition 
Programs 


Paul  Reinhart  ■  Brian  Vanyo 


Accelerating  advancements  in  threat 
military  capabilities  are  driving  sweep¬ 
ing  changes  in  acquisition  intelligence 
support.  Defense  acquisition  and  intel¬ 
ligence  leaders  recognize  that  greater 
consideration  of  threat  capabilities  throughout  a 
program's  acquisition  life  cycle  can  reduce  devel¬ 
opmental  cost  and  operational  risk.  To  facilitate 
threat-informed  acquisition  decisions,  the  defense 
intelligence  community  overhauled  its  threat  sup¬ 
port  products  and  procedures  in  2016.  This  article 
discusses  these  reforms  and  the  outlook  for  im¬ 
proved  intelligence  integration. 

In  late  2013,  then  Assistant  Secretary  of  Defense  for  Acquisition 
(ASD[A])  Katrina  McFarland  formed  an  integrated  product  team  (IPT) 
to  improve  acquisition  intelligence  support  by  making  it,  in  her  words, 
"more  agile  to  the  emerging  threat  environment  and  efficient  in  the 
application  of  better  business  practices."  Mrs.  McFarland  sought  more 
rapid  delivery  of  threat  information  to  acquisition  customers  in  order 
to  more  effectively  support  program  requirements  and  design  deci¬ 
sions.  To  that  end,  she  directed  intelligence  producers  and  acquisition 
customers  to  "work  together  to  develop  a  better  way  to  make  threat 
support  more  timely,  relevant  and  dynamic." 

In  a  series  of  meetings  in  early  2014,  the  IPT  studied  existing  threat 
support  products  and  practices  to  develop  recommendations  for 
improvement.  Initial  discussions  focused  on  two  intelligence  prod¬ 
ucts  that  have  traditionally  supported  acquisition  customers:  the 


Reinhart  and  Vanyo  are  Defense  Intelligence  Agency  intelligence  officers  who  special¬ 
ize  in  aviation-related  technologies  and  provide  analysis  of  threats  to  Major  Defense 
Acquisition  Programs.  They  led  many  of  the  integrated  product  team  meetings ,  working 
groups ,  and  pilot  programs  that  influenced  recent  threat  support  reforms. 
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Capstone  Threat  Assessment  (CTA),  which  forecast  threat 
capabilities  in  specific  warfare  domains;  and  the  System 
Threat  Assessment  Report  (STAR),  which  supported  a  spe¬ 
cific  defense  acquisition  program  by  identifying  all  relevant 
threats  and  their  forecast  capabilities  at  program  initial  oper¬ 
ational  capability  (IOC)  and  10  years  after.  The  IPT  identified 
many  problems  associated  with  CTA  and  STAR  development, 
including  the  following: 

Insufficient  customer  support:  CTA  and  STAR  updates 
occurred  biennially  with  little  input  from  customers  on  de¬ 
sired  content.  Regardless,  their  lengthy  production  period 
precluded  timely  response  to  any  expressed  intelligence 
needs.  Moreover,  Department  of  Defense  (DoD)  policy  did 
not  require  STAR  production  to  support  early  program-de¬ 
sign  decisions  that  greatly  impact  program  cost. 

Limited  threat  forecasting:  CTA  and  STAR  threat  sections 
were  sometimes  more  historical  than  predictive  and  conse¬ 
quently  of  limited  value  to  acquisition  customers,  who  often 
desire  20-year  threat  projections  due  to  the  lengthy  span  of 
time  between  program  initiation  and  IOC. 

Redundant  production:  Many  CTAs  and  STARs  contained 
similar  threat  sections  that  intelligence  analysts  had  to  sepa¬ 
rately  write  and  review,  placing  a  huge  production  burden  on 
limited  analytic  resources  in  the  Department  of  Defense  Intel¬ 
ligence  Community  (DoDIC). 

Analytic  inconsistencies:  Because  each  CTA  and  STAR  was 
produced  individually  on  different  timelines,  they  sometimes 
contained  slight  differences  in  analysis. 

Lack  of  analytic  prioritization:  CTA  and  STAR  production  is 
federated  among  various  intelligence  support  offices,  so  the 
DoDIC  as  an  enterprise  has  little  insight  into  threat  topics  that 
affect  the  most  critical  or  greatest  number  of  defense  acquisi¬ 
tion  programs.  As  a  result,  the  Defense  Intelligence  Agency 
(DIA)  could  not  influence  analytic  prioritization  to  better  sup¬ 
port  acquisition  needs. 

Proposed  Intelligence  Reforms 

To  correct  these  threat  support  problems,  the  IPT  recom¬ 
mended  intelligence  reforms  aimed  to  reduce  production  in¬ 
efficiencies,  thereby  enabling  increased  levels  of  intelligence 
support  throughout  a  program's  life  cycle.  The  I PT  specifically 
proposed  replacing  the  CTA  with  a  new,  centrally  managed 
threat  library  and  replacing  STARs  with  a  new  product  called 
the  Validated  Online  Lifecycle  Threat  (VOLT)  report. 

The  Defense  Intelligence  Threat  Library  is  a  collection  of  ap¬ 
proximately  300  threat  assessments,  known  as  threat  mod¬ 
ules,  each  of  which  addresses  threat  capability  developments 
in  the  next  20  years  in  a  specific  topic  area,  such  as  electronic 
warfare;  air-to-air  missiles;  early  warning  radars;  laser  weap¬ 
ons;  cyberwarfare;  and  adversary  tactics.  Authored  by  Sci¬ 
ence  and  Technical  Intelligence  (S&TI)  experts  in  the  DoDIC, 


threat  modules  provide  broad  threat  technology  trends  and 
projections  with  hyperlinks  to  supporting  foundational  intel¬ 
ligence.  DIA's  Defense  Technology  and  Long-Range  Analysis 
Office  centrally  manages  the  production  and  update  of  threat 
modules  to  ensure  their  currency  and  availability  for  use  as 
the  primary  threat  content  in  VOLT  reports.  Module  updates 
occur  on  schedule  approximately  every  2  years  or  as  needed 
to  account  for  significant  threat  developments. 

A  VOLT  report  is  the  primary  threat  document  supporting 
a  specific  defense  program.  It  includes  all  threat  modules 
deemed  relevant  to  the  supported  program  and  explains 
their  relevance  to  program  requirements.  The  VOLT  report 
also  answers  specific  intelligence  questions  or  requests  raised 
by  the  program,  capability  developer,  or  test  representative. 
VOLT  delivery  includes  a  static  record  copy  of  threat  as  well 
as  a  dynamic  report  that  is  hosted  on  classified  intelligence 
networks  and  embeds  only  the  most  current  threat  modules. 
Therefore,  when  a  single  module  is  updated,  all  VOLTs  using 
it  are  updated  online  simultaneously  to  guarantee  that  all  af¬ 
fected  programs  have  access  to  the  most  current  threat  infor¬ 
mation  at  the  same  time. 

The  IPT  perceived  many  benefits  of  such  reform.  Central¬ 
ized  management  of  all  threat  projections  would  improve 
analytic  resource  management  and  eliminate  redundant  and 
sometimes  inconsistent  CTA  and  STAR  production  efforts. 
Intelligence  support  offices  traditionally  responsible  for  STAR 
production  would  be  able  to  assemble  VOLTs  much  faster  and 
earlier  in  the  acquisition  cycle  using  validated  threat  content 
maintained  in  the  Threat  Library.  Moreover,  S&TI  experts 
would  have  more  time  to  devote  to  futures  analysis  in  threat 
module  production  to  improve  the  overall  quality  of  acquisi¬ 
tion  intelligence  support.  Their  assignment  to  produce  specific 
threat  modules  would  also  enable  more  direct  feedback  from 
acquisition  customers  to  improve  these  products.  Finally,  the 
dynamic  design  of  VOLT  updates  would  offer  programs  instan¬ 
taneous  awareness  of  threat  developments  to  quickly  inform 
acquisition  decision  making. 

Although  these  proposed  reforms  were  conceptually  prom¬ 
ising,  three  pilot  programs  between  June  2014  and  March 
2015  verified  their  actual  value.  VOLT  delivery  was  achieved 
in  a  fraction  of  the  time  ittookto  publish  a  STAR.  New  VOLT 
processes  drove  closer  interaction  and  increased  communi¬ 
cation  between  the  VOLT  author  and  acquisition  customer  to 
improve  product  relevance.  S&TI  experts  reported  spending 
much  less  time  on  threat  module  production  than  they  typi¬ 
cally  spent  on  CTA  and  STAR  reviews  in  a  2-year  timeframe. 
Finally,  the  intelligence  community  demonstrated  that  dy¬ 
namic  VOLTs  could  support  all  programs,  even  those  requir¬ 
ing  special  DoD  clearances. 

Threat  Support  Reform  Implementation 

The  pilot  programs'  positive  findings  ultimately  led  to  a  formal 
action  for  reform.  In  his  April  2015  Better  Buying  Power  3.0  Im¬ 
plementation  Directive,  former  Under  Secretary  of  Defense  for 


Defense  AT&L:  May-June  2017 


18 


Acquisition,  Technology,  and  Logistics  Frank  Kendall  ordered 
the  ASD(A)  to  partner  with  the  Services  and  the  intelligence 
community  to  "develop  a  plan  for  reducing  latency  and  im¬ 
proving  intelligence  data  integration  through  transition  to  the 
Validated  Online  Lifecycle  Threat  (VOLT)  and  Threat  Library." 

Since  then,  DIA  has  taken  a  number  of  actions  to  facilitate  the 
expeditious  transition  to  Threat  Library  and  VOLT.  It  trans¬ 
formed  existing  CTAs  into  threat  modules  to  accelerate  Threat 
Library  development  and  to  insure  against  late  module  delivery 
by  S&TI  experts.  DIA  began  tasking  S&TI  experts  to  produce 
threat  modules  in  December  2015  and  is  validating  those  mod¬ 
ules  upon  completion.  Throughout  2015  and  2016,  DIA  hosted 
many  working  groups  of  acquisition  and  intelligence  profession¬ 
als  to  refine  threat  module  and  VOLT  content  and  process  re¬ 
quirements.  These  meetings  shaped  recent  changes  to  policies 
governing  threat  support  (DIA  Instruction  5000.002,  updated 
on  Sept.  19,  2016,  and  DoD  Instruction  5000.02,  updated  on 
Jan.  26, 2017)  to  officially  replace  STARs  and  CTAs  with  VOLTs 
and  the  Threat  Library.  Finally,  in  late  2016  DIA  began  devel¬ 
oping  the  Threat  Library  System,  which  will  host  all  published 
threat  modules  and  VOLTs  on  classified  intelligence  networks, 
assist  in  dynamic  module  and  VOLT  production,  and  enable 
improved  customer  feedback;  expected  completion  is  late  2017. 

Until  the  Threat  Library  System  is  completed,  DIA  is  managing 
the  Threat  Library  on  classified  collaborative  editing  websites. 
By  the  end  of  2016,  the  library  included  more  than  100  expert- 
published  threat  modules,  and  almost  200  new  threat  mod¬ 
ules  will  be  published  in  2017.  For  topics  that  lack  a  published 
threat  module,  DIA  has  identified  approved  alternate  sources 
for  use  in  VOLTs. 

In  January  2017,  Service  intelligence  offices  began  producing 
VOLTs  in  place  of  STARs,  and  their  anticipated  benefits  are 
being  realized.  VOLT  authors  have  provided  threat  support 
within  90  days,  and  in  some  cases  within  30  days,  to  suffi¬ 
ciently  meet  acquisition  event  timelines.  The  threat  content 
in  published  VOLTs  is  more  current  than  traditional  STAR 
content— by  more  than  1  year  on  average.  And  because  a 
VOLT's  relevant  threat  modules  are  continually  updated,  ac¬ 
quisition  customers  no  longer  have  to  wait  2  years  or  more 
(the  average  STAR-update  rate)  before  being  notified  of 
threat  developments. 

Customer-Driven  Production 

The  biggest  change  with  new  threat  support  procedures  is 
that  acquisition  customers  truly  drive  VOLT  production.  DoD 
policy  now  mandates  VOLT  support  at  the  Materiel  Develop¬ 
ment  Decision,  Milestone  A  Decision  Review,  Development 
Request  for  Proposals  Release  Decision  Point,  Milestone  C 
Decision  Review,  and  the  Full-Rate  Production  or  Full  Deploy¬ 
ment  Decision.  But  acquisition  customers  may  request  VOLT 
delivery  to  support  other  events  or  design  decisions  as  needed. 

Customers  initiate  VOLT  production  by  submitting  a  VOLT 
request  form  and  system  documentation  to  their  intelligence 


support  offices.  In  that  request,  customers  communicate  key 
threat  areas  of  interest,  select  desired  levels  of  content  tailor¬ 
ing  and  provide  specific  intelligence  questions  for  the  VOLT 
author  to  answer. 

Customers  have  another  opportunity  to  shape  VOLT  content 
at  a  threat  steering  group  (TSG)  meeting,  which  is  a  formal 
threat  discussion  involving  the  program  manager  and/or  rep¬ 
resentative,  capability  developer,  service  test  representative, 
VOLT  author,  VOLT  validator,  and,  as  applicable,  a  representa¬ 
tive  of  the  Director  of  Operational  Test  and  Evaluation.  This 
meeting  gives  every  TSG  member  the  opportunity  to  comment 
on  the  scope  of  threat  discussed  in  the  VOLT  and  to  make 
other  intelligence  requests  to  ensure  that  the  final  product 
best  serves  each  member's  needs. 

Threat  Support  Reform  Outlook 

When  the  Threat  Library  System  becomes  operational  later 
this  year,  acquisition  customers  will  have  a  single  place  to 
access  all  threat  modules  and  VOLTs.  They  will  be  able  to 
subscribe  to  specific  threat  modules  or  VOLTs  to  receive  no¬ 
tifications  when  these  products  are  updated.  Users  will  also 
have  the  means  to  provide  direct  feedback  to  module  or  VOLT 
authors  and  drive  changes  to  subsequent  product  updates. 

Feedback  is  vital  to  the  success  of  this  effort,  for  it  remains  a 
work  in  progress.  A  working  group  of  Threat  Library  System 
users  will  continue  to  shape  its  requirements  during  develop¬ 
ment.  The  complete  list  of  threat  modules  will  continue  to 
evolve  as  new  modules  are  added  or  existing  topics  are  reor¬ 
ganized.  Acquisition,  requirements  and  test  representatives 
can  help  refine  threat  module  and  VOLT  content  by  engaging 
with  authors  and  the  DIA. 

As  these  threat  support  reforms  mature,  their  benefits  will 
likely  accrue.  The  pilot  programs  already  demonstrated  that 
reforms  will  save  the  DoDIC  thousands  of  hours  of  analytic 
work  each  year  and  give  DIA  leaders  unprecedented  insight 
into  threats  affecting  defense  programs,  enabling  improved 
analytic  prioritization  and  intelligence  enterprise  manage¬ 
ment.  More  importantly,  these  reforms  will  enable  more 
responsive  threat  support  to  better  serve  the  informational 
needs  of  a  variety  of  acquisition  customers  throughout  a 
program's  life  cycle. 

In  today's  rapidly  changing  threat  landscape,  it  is  vital  to  our 
national  security  that  adversary  capabilities  are  considered 
in  every  decision  that  shapes  future  U.S.  defense  capabilities. 
Improved  intelligence  integration  will  give  acquisition  lead¬ 
ers  the  kind  of  agility  they  seek  to  factor  threat  developments 
into  milestone  decisions  alongside  traditional  programmatic 
considerations  such  as  cost,  schedule  and  performance.  Ulti¬ 
mately,  these  reforms  will  arm  DoD  leadership  with  the  best 
intelligence  today  in  their  efforts  to  equip  our  forces  for  the 
emerging  threats  of  tomorrow.  & 

The  authors  can  be  contacted  at  paul.reinhart@dodiis.mil  and  brian. 
vanyo@dodiis.mil. 
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Could  Your 
Organization  Use 

a  Leadership 

Booster  Shot? 


Woody  Spring  ■  John  Larson 


Leadership  is  a  key  ingredient  for  all  orga¬ 
nizations.  In  the  professional  acquisition 
community  and  others  like  it,  functional 
experts  can  quickly  find  themselves  in 
situations  where  they  need  more  lead¬ 
ership  dexterity  when  promoted  to  fill 
key  leadership  positions. 

The  best  leaders  not  only  continue  learning  new  leadership  techniques 
throughout  their  careers;  they  also  develop  and  mentor  leadership  within 
the  organization  to  enable: 

■  Developing  managerial  effectiveness 
■  Inspiring  others 
■  Developing  employees 
■  Leading  a  team 
■  Guiding  change 

■  Managing  internal  stakeholders  and  politics 

More  and  more  leaders  have  begun  making  strategic  investments  in  the 
professional  development  of  their  personnel.  They  know  the  vital  impor¬ 
tance  of  long-range  planning,  which  includes  the  people  variable— one  of 
the  keys  to  an  organization's  long-term  success.  In  the  last  6  years,  the 
Defense  Acquisition  University  (DAU)  has  had  the  good  fortune  to  be  part 
of  their  learning  formulary  through  its  Acquisition  Leadership  Development 


Spring,  a  graduate  of  West  Point  and  former  NASA  astronaut ,  is  a  professor  of  Engineering, 
Test  and  Evaluation  at  the  Defense  Acquisition  University  (DAU)  West  Region  in  San  Diego , 
California ,  as  well  as  chief  learning  officer  and  executive  coach.  Larson,  a  retired  U.S.  Navy 
commander ;  is  a  DAU  professor  of  Systems  Engineering  in  San  Diego.  He  is  Level  III  certified 
in  Life  Cycle  Logistics;  Production ,  Quality  and  Manufacturing ;  Systems  Planning ,  Research , 
Development  and  Engineering  (SPRDE);  and  Program  Management. 


21 


Defense  AT&L:  May-June  2017 


Table  1.  Modules  and  Focus  Areas  Explained 


Module 

Focus  Area 

What  is  Leadership? 

Leadership  attributes  and  their  effectiveness  in  an  acquisition  environment 

Recognizing  Leadership  Differences 

Personality  preferences  (i.e.,  temperaments)  in  the  context  of  your  strengths 
and  challenges 

Recognizing  People's  Differences 

Personality  preferences  of  others  and  how  best  to  work  with  them 

People's  Styles  at  Work 

People's  styles  to  enhance  relationships,  increase  productivity  and  add  richness 

Leading  High-Performance  Teams 

Building  and  leading  teams  that  raise  the  bar  on  performance  standards 

Leading  in  Different  Environments 

Leadership  differences  in  various  acquisition  leadership  environments 

Integrated  Product  Teams  (IPTs) 

Importance  and  challenges  of  IPTs 

Team  Facilitation  Skills 

Facilitating  small  teams  and  developing  associated  competencies 

Four  Roles  of  Leadership 

Four  essential  functions  required  of  successful  leaders 

Change  Leadership 

Dr.  John  Kotter's  eight-step  model  and  Dr.  Kurt  Lewin's  three-step  model 

Coaching  in  a  Changing  Environment 

Reflective  and  effective  learning  and  performance 

Developing  Acquisition  Leaders 

Leadership  tenets  in  the  Department  of  Defense  and  the  guidelines  for  leader¬ 
ship  development  plans 

Workshop  (ALDW)  series.  These  workshops  provide  a  mix¬ 
ture  of  hard-hitting  content  and  practical  exercises  represent¬ 
ing  typical  workplace  challenges. 

ALDW  Explained 

At  its  core,  ALDWs  promote  the  development  of  leadership 
skills  that  the  acquisition  workforce  requires  through  cus¬ 
tomized  training  solutions  based  on  workplace  needs.  Or¬ 
ganizations  can  select  from  a  variety  of  workshop  varia¬ 
tions.  The  sample  modules  along  with  their  focus  areas  are 
outlined  in  Table  1  and  were  distilled  from  a  wide  range  of 
parent  DAU  courses. 


This  "menu"  of  ALDW  options  are  especially  applicable  for 
intact  teams  to  help  them  work  together  to  explore  and  de¬ 
velop  (or  refine)  their  skills.  With  few  exceptions,  each  ALDW 
module  is  a  standalone  topic;  although  many  modules  work 
synergistically  when  integrated  and  embody  a  greater  under¬ 
standing  of  the  leadership  essentials.  ALDWs  can  be  applied 
to  grow  leadership  skills  in  junior  personnel,  or  perhaps  more 
rigorously  help  senior  leaders  explore  complex  and  impending 
leadership  challenges  they  sometimes  face.  Simply  stated,  an 
ALDW  is  a  very  flexible  construct  that  can  address  organiza¬ 
tional  imperatives  at  all  levels,  especially  those  that  pave  the 
way  for  high-performance  teams. 


Table  2.  Acquisition  Leadership  Development  Workshop  Modules 


Leadership  Development 

|  Self  - 

Development 

Skills  for 
Leadership 

Skills  for  Teams 

Stakeholder 

Relationships 

What  is 
Leadership? 

Leading  High- 
Performance 

Teams 

Four  Roles  in 
Leadership 

Emotional 

Intelligence 

Critical  Thinking 

Dealing  with 
Conflict 

Forging  the 
Relationships 

Recognizing 

Leadership 

Differences 

Leading  in  Differ¬ 
ent  Environments 

Change 

Leadership 

Providing 

Effective 

Feedback 

Problem-Solving 

Tools 

Building  Trust 

Stakeholder 

Engagement 

Plan 

Recognizing 

People's 

Differences 

Integrated  Product 
Teams 

Coaching  in  a 

Changing 

Environment 

Mentoring 

Communication 

Strategies 

Accountability  & 
Empowerment 

Stakeholder 

Communication 

Plan 

People's 

Styles  at 

Work 

Team  Facilitation 

Developing 

Acquisition 

Leaders 

Declaring  an 

Extraordinary 

Future 

Dale  Carnegie: 
Influencing  People 

Groupthink 
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Table  3.  Various  Self-Examination  and  Evaluation  Modules 


Name 

Focus 

Emotional  Intelligence 

Learning  how  to  know  yourself  and  read  other's  emotions 

Effective  Feedback 

Delivering  feedback  using  various  techniques 

Mentoring 

Declaring  an  Extraordinary  Future 

Critical  Thinking 

Problem  Solving  Tools 

Teaching,  training  and  enabling  the  workforce  through  mentoring 

Choosing  to  exceed  a  predictable  future;  done  as  part  of  executive  coaching 

Learning  to  think  clearly  and  completely;  tools  and  techniques 

Various  tools  for  group  problem  solving  and  issue  resolution 

Communication  Strategies 

Various  ways  to  get  the  message  across  and  connect  with  others 

Influencing  People 

Various  techniques  to  influence  stakeholders  and  others 

Meyers  Briggs  (Personality)  Type  Indica¬ 
tor 

A  self-awareness  of  individual  personality  preferences  and,  by  extension,  an  under¬ 
standing  that  other  people  have  different  preferences 

ALDWs  can  start  with  a  discussion  on  leading  up,  down  and 
across,  or  ignite  the  participants'  interest  with  a  critical  think¬ 
ing  challenge.  No  matter  what  the  facilitator  uses  at  the  start¬ 
ing  gate,  the  participants  quickly  learn  more  about  the  deeper 
context  of  leadership  and  the  extensive  tools  that  are  at  their 
disposal.  To  enrich  the  workshop  experience  throughout  an 
ALDW,  DAU  facilitators  draw  from  their  technical  knowledge 
and  personal  experiences  to  reinforce  the  lessons  learned 
from  similar  leadership  challenges  they  faced.  ALDW  partici¬ 
pants  need  not  be  employed  in  acquisition-coded  positions 
to  attend  the  workshop.  Unlike  other  training  solutions,  there 
are  no  prerequisites— just  two  expectations  that  participants 
need  to:  (1)  approach  these  workshops  with  open  minds  and 
(2)  demonstrate  a  willingness  to  think  more  about  their  own 
thinking  (i.e.,  metacognition)  as  they  learn  how  to  put  proven 
methods  into  action. 

Table  2  shows  how  ALDW  modules  can  be  aligned  for 
growth  and  alignment  within  intended  focus  areas  (e.g., 


self-development,  skills  for  leadership,  skills  for  teams,  and 
stakeholder  relationships). 

These  delivery  combinations  can  also  include  modules  that 
focus  on  self-examination  and  evaluation— all  key  tools  that 
promote  more  self-awareness  such  as  shown  in  Table  3. 

Since  intact  teams  very  often  face  challenges  (and  future  op¬ 
portunities)  that  require  agreement,  the  delivery  combinations 
can  include  modules  that  help  teams  break  through  various 
decision  logjams  characteristic  of  some  workplace  challenges: 

■  Dealing  with  Conflict 

■  Building  Trust 

■  Accountability  and  Empowerment 

■  Groupthink 

In  a  more  magnified  view,  Figure  1  shows  how  a  module  fo¬ 
cused  on  building  (or  restoring)  trust  can  be  addressed  in  a 
customized  ALDW.  The  resultant  practical  exercise  includes 


Figure  L  Logical  Plan  Building  Toward  Action 


Modules 

What  is  Leadership 

4 

.  Problem  Solving 

4 

Building  Trust 
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•  Change  Leadership 
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4 

<U 
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•  Using  tools, 

•  Understand  steps 

•  Develop  leadership 

5 
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to  build/rebuild 

strategy  and  plan  to 
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leadership  issue 
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•  Plan  is  to  be  left  behind 
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& 
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a  structured  activity  model  that  drives  to  the  root  cause  of  a 
known  case  of  mistrust  followed  by  the  steps  and  strategy 
to  restore  trust.  After  the  participants  complete  a  rigorous 
exercise,  their  commander  is  asked  to  join  the  workshop  for  a 
decision  brief,  which  includes  a  review  of  their  strategic  plan  to 
guide  the  workforce  and  specific  performance  targets  to  verify 
the  participants  achieved  their  targets.  The  participants  take  a 
lot  of  pride  in  their  plans,  and  show  a  great  eagerness  to  hear 
their  commander  say,  "Make  it  so." 

Stakeholder  management  has  become  more  and  more  crucial 
in  promoting  leadership  gains,  and  DAU  developed  a  series  of 
workshops  that  address  stakeholder  management,  including: 

■  Forging  Stakeholder  Relationships 

■  Stakeholder  Engagement  Plan 

■  Stakeholder  Communication  Plan 

What  Can  You  Expect  in  the  Workshop? 

In  addition  to  our  commitment  to  best  meet  the  needs  of 
acquisition  professionals,  DAU  provides  certified  and  sea¬ 
soned  facilitators.  Our  facilitators  help  participants  col¬ 
lectively  understand  their  risks  and  issues,  investigate  the 
solution  space  and  develop  actionable  recommendations. 
This  last  item  is  all  about  ownership  and  leaves  the  intact 
team  with  the  responsibility  to  implement  the  actions  to 
achieve  the  intended  outcomes.  As  the  participants  craft 


■  Howard  Goldman,  Commander  Naval  Undersea  Warfare 
Center  Division,  Newport,  Rhode  Island: "...  developing  the 
stakeholder  plan  was  valuable  in  that  it  forces  you  to  give  a 
thorough  examination  of  who  your  stakeholders  are  and  what 
their  interests  are.  The  DAU  instructors  were  outstanding." 
The  real  value  of  the  training  will  be  in  our  follow-up  and  ex¬ 
ecution  of  the  four  communication  plans  that  were  initiated 
during  the  workshop. 

■  Student  comments  from  recent  2-day  workshop: 

"This  is  a  very  insightful  class. ..able  to  use  the  tools... ." 
"VERY  informative  class!  Well  worth  my  time" 

"Material  is  very  relevant  in  a  team  environment." 

"Relevant.  Beneficial.  Informative.  FUN!" 

"Very  useful  and  I  will  recommend  to  others." 

"Outstanding  course." 

■  Acquisition  Manager,  U.S.  Pacific  Fleet:  "DAU  has  been 
an  excellent  resource  for  me  and  together,  we  have  ensured 
hundreds  of  Navy  and  Joint  personnel  have  been  able  to  take 
classes  in  preparing  performance  work  statements.  We  will 
never  know  the  true  impact  of  all  that  training  but  I  have  seen 
the  results  in  multiple  performance  work  statements  prepared 
by  graduates  of  your  PWS  classes.  I've  seen  first-hand  how 
much  more  confident  the  employees  are  in  preparing  their 
requirements." 


As  the  participants  craft  their  own 
heading,  they  also  become  ideally 
suited  to  serve  as  the  actions' 
champions  back  in  the  workplace. 


their  own  heading,  they  also  become  ideally  suited  to  serve 
as  the  actions'  champions  back  in  the  workplace.  Something 
else  happens  at  these  workshops  that  is  less  apparent  at 
the  beginning:  After  the  first  couple  of  hours,  the  partici¬ 
pants  not  only  are  exposed  to  some  very  relevant  material 
but  form  conspicuous  bonds  within  the  team  and  can  serve 
as  the  guiding  coalition  across  the  organization  to  see  the 
required  changes  through  to  their  conclusions. 

What  Past  Participants  Have  to  Say 

If  testimonials  indicate  the  usefulness  of  ALDWs,  the  com¬ 
ments  below  reinforce  the  reports  of  its  efficacy.  More  im¬ 
portant,  what  measurable  impacts  have  these  ALDWs  had? 
Note  that  many  customers  were  very  grateful  but  preferred 
not  to  be  quoted.  The  following  statements  were  made  by  a 
representative  group  that  volunteered  to  be  quoted: 


■  PEO  C4I  Tactical  Networks  Program  Office  (PMW 160): 
DAU  provided  PMW  160  leadership  an  independent  and 
objective  assessment  of  the  CANES  [Consolidated  Afloat 
Networks  and  Enterprise  Services]  program  and  organiza¬ 
tion.  Additionally,  the  effort  provided  the  PM  [program  man¬ 
ager]  and  DPM  [deputy  program  manager]  actionable  orga¬ 
nizational  and  related  process  improvements  to  enable  more 
successful  execution  for  CANES.  Impact:  Mr.  Jim  Churchill 
(Acting  PM)  commented  the  Independent  Program  Review 
was  "pure  gold"  and  asked  for  DAU  West  to  support  future 
efforts  to  implement  recommendations  with  the  PMW  160 
Leadership  team. 

■  Naval  Weapons  Station,  Seal  Beach,  California  (one  of 
many  customers  who  seek  this  workshop  series):  The  ALDW 
helped  rectify  a  corrective  action  by  bringing  shop  workers, 
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Depending  on  your  need,  we  may 
recommend  an  organizational 
assessment  survey  tailored  to  your 
organization  which  is  intentionally 
designed  to  uncover  any  issues. 


foremen  and  supervisors  together  to  explore  leadership, 
accountability  and  building  (restoring)  trust,  as  well  as  prob¬ 
lem-solving  tools  for  use  by  all  personnel. 

■  RADM  David  Lewis,  Space  and  Naval  Warfare  Systems 
Commander:  "We've  seen  an  increase  in  critical  thinking.  An 
example  is  DCGS-N. ...  Two  years  ago,  we  saw  classic  acquisi¬ 
tion  thought  for  a  monolithic  5-year  program  to  get  capability 
out  to  the  Fleet.  Now  we  are  fielding  full  capability  builds  every 
18  months.  We've  also  seen  deeper  thought  into  CANES  inte¬ 
gration  issues  among  the  variety  of  platforms.  DAU's  training 
in  this  area  has  helped." 

■  Col  Ryan  Britton,  former  ICBM  Program  Director,  Air  Force 
Nuclear  Weapons  Center:  "DAU  conducted  a  series  of  Ac¬ 
quisition,  Risk  Management  and  Source  Selection  workshops 
that  focused  on  tackling  the  major  issues  including  workforce 
competencies,  and  acquisition  and  program  strategies  that 
could  interfere  with  program  achievements.  I  planned  for  a 
9-month  protest— didn't  materialize— attributed  to  DAU's 
support.  These  workshop  series  were  well  received  and  di¬ 
rectly  resulted  in  improved  acquisition  outcomes." 

Author  Spring’s  Leadership  Retrospect 

As  a  former  Army  officer,  whether  a  platoon  leader  in  Vietnam, 
an  Experimental  Test  Pilot  team  lead,  a  major  acquisition  pro¬ 
gram  manager,  a  NASA  astronaut,  or  a  division  manager  with 
industry,  I  have  experienced  the  requirement  for  continuous 
growth  in  different  types  of  leadership  and  the  need  for  various 
tools  and  techniques  for  both  my  personal  success  and  that 
of  my  programs  in  different  environments.  In  many  instances, 
I  wish  I  could  have  had  an  organization  like  DAU  available  to 
me  to  help  facilitate  the  issues  and  challenges  my  programs 
and  I  faced  on  our  journey  to  success.  Leading  programs— to 
be  able  to  function  without  you,  mentoring  subordinates  to  be 
leaders,  enabling  organizational  innovation,  trust,  and  respon¬ 
sibility  and  leaving  programs  in  better  condition  to  succeed 
than  when  you  assumed  the  leadership  are  universal  lead¬ 
ership  challenges.  John  Larson,  this  article's  co-author,  and 
I  thoroughly  enjoy  our  current  jobs  as  DAU  professors  and 
our  involvement  with  Mission  Assistance  workshops  in  the 
workplace  and  at  the  point  of  need.  We  are  not  alone;  every 
DAU  Mission  Assistance  facilitator  feels  the  same  way. 


Summary 

Does  your  organization  require  a  shot  of  energy  to  gain  the 
learning  lift  it  needs?  With  a  booster  shot  of  the  leadership 
"knowhow,"  found  in  our  Acquisition  Leadership  Develop¬ 
ment  Workshops,  organizations  can  achieve  noticeable  per¬ 
formance  gains  with  greater  efficiency  and  effectiveness.  How 
can  we  help  you?  Visit  the  DAU  website  at  www.dau.mil  or 
contact  the  DAU  Business  Unit  in  your  region  to  learn  more. 
Depending  on  your  need,  we  may  recommend  an  organiza¬ 
tional  assessment  survey  tailored  to  your  organization  which 
is  intentionally  designed  to  uncover  any  issues  (See  Spring's 
article— "Take  a  Deep  Dive  with  DAU"  in  the  January-February 
2017  issue  of  Defense  AT&L  magazine).  Based  on  survey  re¬ 
sults,  DAU  would  then  have  enough  objective  data  to  help 
you  determine  the  most  appropriate  workshop  agenda  that 
draws  from  the  modules  described  above.  Contact  us  for  your 
leadership  inoculation,  today.  & 

The  authors  can  be  contacted  at  woody.spring@dau.mil  and  john.larson@ 
dau.mil. 


MDAP/MAIS  Program 
Manager  Changes 


With  the  assistance  of  the  Office  of  the  Secretary  of  De¬ 
fense,  Defense  AT&L  magazine  publishes  the  names  of  in¬ 
coming  and  outgoing  program  managers  for  major  defense 
acquisition  programs  (MDAPs)  and  major  automated  in¬ 
formation  system  (MAIS)  programs.  This  announcement 
lists  the  one  such  change  of  leadership  reported  for  both 
civilian  and  military  program  managers  for  the  months  of 
January  and  February  2017. 

Navy/Marine  Corps 

COL  David  C.  Walsh  relieved  COL  Steven  R.  Girard  as 

program  manager  for  the  H-1  Upgrades  (4BW/4BN)  pro¬ 
gram  in  PMA  276  on  Feb.  14. 
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Value  Adjusted  Total  Evaluated  Price  Could  Be  the  Answer 


Scott  Gilbreth 


Most  people  can  see  the  value  of  evaluating  a  contractor  on  the  life- 
cycle  cost  (LCC)  of  their  solution  instead  of  just  the  initial  purchase 
price.  In  some  form  or  fashion,  we  use  a  life-cycle  perspective  for  large 
purchases  in  our  personal  life.  A  typical  diesel  engine  in  a  truck  has  a 
longer  service  life,  greater  pulling  power  and  fuel  efficiency,  and  retains 
its  value  longer  than  a  gasoline  engine.  But  diesels  typically  require  more  routine 
maintenance,  and  their  repairs  and  the  fuel  itself  are  more  expensive.  That  sounds 
like  a  spreadsheet  waiting  to  happen! 

LCC  as  a  concept  has  many  fans,  but  in  practice  has  become  undesirable  in  Department  of  Defense  (DoD)  source 
selections.  A  source-selection  team  that  wishes  to  use  LCC  or  the  most  probable  life-cycle  cost  (MPLCC)  for  cost 
contracts  in  its  source-selection  criteria  will  run  into  a  great  deal  of  resistance. 


Gilbreth  is  a  professor  of  Contract  Management  at  the  Defense  Acquisition  University's  Midwest  Region  in  Kettering ,  Ohio ,  and  recently  com¬ 
pleted  a  short-term  assignment  with  source-selection  advisers  from  the  Acquisition  Center  of  Excellence  at  Wright-Patterson  Air  Force  Base. 
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Experts  I  spoke  with  from  both 
the  Navy  and  Army  saw  what 
the  Air  Force  went  through 
and  have  had  their  own  "ctyse 
calls"  with  LCC  protests. 

r  N 

a 

I  asked  seasoned  cost  estimators  and  source-selection  experts 
from  each  military  Service  to  help  me  understand  why  there  is 
such  opposition  to  LCC  evaluations.  Their  general  consensus 
was  surprising  and  the  details  of  each  conversation  were  strik¬ 
ingly  similar.  Each  expert  addressed  some  variation  of  these 
longstanding  issues: 

■  Life-cycle  cost,  by  definition,  will  occur  well  beyond  the 
life  of  an  individual  contract. 

■  The  government  would  be  selecting  a  contractor  based 
on  its  estimate  of  future  sustainment  cost  even  though  it 
can't  be  held  accountable  for  the  validity  of  the  estimate. 

■  The  operational  life  and  operational  profile  of  a  system 
decades  into  the  future  is  wildly  unpredictable. 

■  And  20  or  30  years  of  theoretical  sustainment  cost  over¬ 
shadows  current,  real  procurement  cost. 


has  taken  the  more  formal  route  with  a  requirement  in  the  Air 
Force  Federal  Acquisition  Regulation  Mandatory  Procedures 
for  approval  at  the  level  of  the  Secretary  of  the  Air  Force- 
Acquisition  (SAF-AQ)  before  MPLCC  can  be  incorporated 
in  source  selections.  The  current  plight  of  LCC  evaluations  is 
interesting,  but  it  becomes  compelling  in  view  of  the  fact  that 
LCC  was  commonly  used  in  government  source  selections  for 
more  than  50  years. 

Life-cycle  cost  considerations  became  fully  integrated  in  DoD 
acquisitions  in  the  late  1950s.  In  a  1979  decision  (B-192488), 
GAO  reiterated,  "We  have  consistently  upheld  life-cycle  cost¬ 
ing,  stating  that  it  is  logical  to  consider  total  anticipated  costs, 
ratherthan  merely  purchase  price."  There  have  been  numerous 
dismissed  and/or  denied  protests,  and  a  few  sustained,  involv¬ 
ing  LCC  over  the  years.  I  couldn't  always  determine  whether 
these  historical  cases  involved  a  traditional  LCC  evaluation 
or  just  a  selected  element(s)  from  Operations  and  Support 
(O&S)  cost.  A  traditional  LCC  evaluation  would  closely  align 
with  most  of  the  LCC  categories:  research  and  development, 
investment,  O&S,  and  disposal  costs.  But  a  modified  LCC  eval¬ 
uation  might  just  evaluate  a  few  of  the  six  lower-level  elements 
within  O&S:  manpower,  operations,  maintenance,  sustainment 
support,  system  improvement  and  indirect  support. 

Either  way,  the  GAO  language  from  the  2007/2008  protests 
reiterated  comments  I  found  in  many  historical  LCC  protests: 
"the  evaluation  of  sustainment  costs  must  be  tailored  to  each 
offerors  approach"  and  "hypothetical  life  cycle  costs  must 
have  reasonable  support."  Regardless  of  how  you  interpret 
the  details  of  either  the  CSAR-X  or  KCX  protest,  it  does  appear 
that  they  represented  the  "last  straw"  for  LCC  as  a  common 
component  of  DoD  source  selections. 


Eventually,  either  the  Air  Force's  CSAR-X  (Combat  Search 
and  Rescue  Replacement  Vehicle)  or  KCX  (refueling  tanker) 
protests  from  2007/2008  came  into  the  conversation.  Al¬ 
though  these  protests  encompassed  multiple  issues,  LCC  was 
a  significant  component  in  the  Government  Accountability 
Office  (GAO)  decisions.  The  legal  and  political  fallout  from 
those  protests  appears  to  have  impacted  acquisition  decision 
making  well  beyond  the  Air  Force.  Experts  I  spoke  with  from 
both  the  Navy  and  Army  saw  what  the  Air  Force  went  through 
and  have  had  their  own  "close  calls"  with  LCC  protests. 

It  is  relevant  to  understand  that  DoD  began  a  formal  standard¬ 
ization  effort  for  department-wide  source-selection  proce¬ 
dures  in  the  same  general  time  of  late  2008.  Source  selection 
centers  of  excellence,  designed  to  promulgate  best  practices 
within  each  of  their  Services,  have  become  more  engaged  with 
their  counterparts  in  other  Services.  But  while  the  experts  I 
spoke  with  completely  agree  with  the  policy  of  discouraging 
LCC  in  evaluations,  their  respective  Services  have  addressed 
the  matter  differently.  For  instance,  there  is  a  Navy  document, 
now  widely  referenced  throughout  DoD,  that  presents  a  deci¬ 
sion  tree  to  help  identify  those  limited  situations  where  LCC 
would  be  appropriate  in  the  evaluation  criteria.  The  Air  Force 


Major  source  selections  within  DoD  today  take  well  more 
than  a  year.  The  number  of  reviews  and  the  level  of  scrutiny 
are  overwhelming.  There  are  military  leaders,  civil  servants 
and  politicians  speaking  to  that  issue— I'll  leave  it  to  them. 
But  in  this  environment,  protests— successful  or  unsuccess¬ 
ful— just  add  to  the  timeline  and  further  delay  getting  re¬ 
quirements  on  contract.  A  Total  Evaluated  Price  (TEP)  full  of 
theoretical  LCCs  can  swing  drastically  by  a  change  in  service 
life  estimate  and  time  value  of  money  adjustments.  This  will 
always  be  an  obvious  target  for  protesters  unhappy  with  the 
evaluation  outcome. 

Source  selection  experts  continually  push  to  limit  complex¬ 
ity  and  challenge  teams  to  select  only  those  evaluation  cri¬ 
teria  that  will  truly  discriminate  between  offerors.  A  look  at 
all  the  major  components  of  LCC  definitely  adds  complexity, 
and  it's  unlikely  that  each  element  is  a  discriminator  between 
offerors.  The  experts  have  a  strong  argument  here!  Is  there 
value  in  the  LCC  perspective,  as  I  believe  there  is?  Does  the 
current  acquisition  environment  makes  traditional  LCC  too 
risky,  which  is  the  experts'  consensus  view?  Given  all  that,  can 
the  trade-off  technique  called  Value  Adjusted  Total  Evaluated 
Price  (VATEP)  provide  a  new  platform  for  LCC? 
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The  2016  DoD  Source  Selection  Procedures  didn't  change  dra¬ 
matically  from  the  2011  version,  other  than  to  formally  intro¬ 
duce  VATEP  trade-off.  This  technique  allows  the  government 
to  place  a  value  on  performance/capabilities  that  exceeds  the 
minimum  established  by  the  requirement.  The  document  goes 
on  to  illustrate  a  higher-priced  offeror  receiving  a  price  decre¬ 
ment  and  becoming  the  low  offeror,  for  evaluation  purposes, 
by  submitting  a  proposal  responding  to  a  stated  objective 
above  the  minimum  requirement.  So  although  the  contractor 
has  a  proposed  price,  that  price  is  reduced  during  evaluation 
by  a  value  attached  to  achieving  such  specific  objective(s). 

Another  illustration  in  the  VATEP  appendix  even  shows  a 
decrement  value  attached  to  receiving  a  "low"  risk.  So,  why 
not  associate  a  decrement  value  for  meeting  LCC  criteria? 
A  contractor  can  only  protest  the  value  of  the  decrement 
prior  to  award,  and  the  standard  generated  from  GAO  com¬ 
ments  only  requires  the  decrement  value  to  be  "reasonably 
supported  by  data  or  analysis."  GAO  also  expects  all  LCC 
evaluations  to  be  tailored  in  accordance  with  each  proposed 
solution.  That  requirement  can  be  satisfied  by  the  applica¬ 
tion  of  decrement  criteria.  Therefore,  I'm  suggesting  that  the 
VATEP  approach  can  provide  source-selection  teams  with 
the  LCC  construct  they  desire  while  greatly  minimizing  the 
complexity  and  risk. 

Just  utilizing  the  VATEP  trade-off  technique  doesn't  alleviate 
all  the  LCC-associated  challenges.  It  will  be  critically  important 
that  those  discriminating  element(s)  of  LCC  be  identified  in 
order  to  use  them  in  a  VATEP  calculation.  Legacy  systems, 
DoD  experts,  and  external  market  research  efforts  can  all  help 
identify  the  key  components.  Data  also  will  be  required  to  pro¬ 
vide  reasonable  support  for  a  decrement  value  associated  with 
each  LCC  attribute.  Historical  data  can  be  analyzed  from  the 
Visibility  and  Management  of  Operating  and  Support  Costs 
system;  both  a  predecessor  system  and  an  analogous  system 
can  be  referenced.  There  is  a  strong  theoretical  case  for  lever¬ 
aging  VATEP  with  LCC.  But  only  over  time  and  with  acquisition 
programs  that  are  willing  to  incorporate  LCC  elements  into 
their  source  selections  will  we  truly  know  its  viability.  In  the 
interim,  let's  examine  a  hypothetical  scenario  to  see  how  LCC 
can  be  integrated  into  a  VATEP  source  selection. 

First  and  foremost,  for  LCC  to  be  relevant  in  a  source  selection, 
market  research  must  reveal  more  than  one  distinct  solution  to 
a  requirement— with  each  solution  having  different  LCC  impli¬ 
cations.  DoD  has  an  ongoing  need  for  long-lasting  (72  hours), 
lightweight  rechargeable  batteries  to  power  the  technology  that 
dismounted  troops  carry  into  battle.  A  quick  summary  of  the 
current,  proven  technology  is  that  ballistic-rated  lithium-ion 
rechargeable  batteries,  providing  150  Watt  hours  for  up  to  36 
hours  at  a  weight  of  1.18  kilograms  (2.6  pounds),  are  built  in  a 
flexible  casing  with  standard  ballistic  plate  dimensions  to  be 
integrated  in  body  armor  vests.  Several  other  battery  technolo¬ 
gies,  labeled  as  advanced  and  post-lithium  batteries,  might  meet 
or  exceed  the  current  government  requirement.  There  are  obvi¬ 
ous  technical  performance  objectives  in  this  requirement  that 


could  be  translated  into  evaluation  decrements  as  illustrated  in 
the  2016  Source  Selection  Procedures.  But  what  are  the  poten¬ 
tial  LCC  considerations?  Well,  we  could  research  the  cost  impli¬ 
cations  of  those  potential  solutions  proving  incompatible  with 
the  existing  Soldier  Worn  Integrated  Power  Equipment  System 
(SWIPES)  orthefast  rechargingand  solar-powered  recharging 
systems.  What  about  the  value  of  a  battery  with  an  extended 
service  life,  maintenance  requirements,  or  disposal  costs  associ¬ 
ated  with  a  complex  chemistry?  These  LCC  based  questions, 
while  related  to  performance  considerations,  are  unique  and 
should  be  separately  valued  under  VATEP. 

Isolating  and  identifying  the  discriminating  elements  between 
various  solutions  is  the  next  step  in  developing  a  LCC  decre¬ 
ment.  A  traditional  LCC  estimate  would  include  end-of-life 
disposal  costs.  Market  research  indicated  that  the  disposal 
concerns  as  well  as  the  minimal  maintenance  requirements 
for  the  various  rechargeable  batteries  were  not  proven  to  be 
discriminators.  Likewise,  compatibility  with  SWIPES  proved 
easily  achievable  regardless  of  battery  type.  The  remaining 
considerations  were  service  life  and  compatibility  with  existing 
fast  rechargers  and  solar  rechargers.  The  decrement  criteria 
for  service  life  could  read  as  follows: 

$x, xxx, xxx  service  life  decrement  for  each  increment  of  100  dis¬ 
charge  cycles  above  the  500  cycle  threshold  point  while  main¬ 
taining  a  charge  capacity  less  than  75  percent  of  the  original 
capability.  Not  To  Exceed  $xx,xxx,xxx  total  decrement  value. 

Government  experts  and  industry  feedback  led  to  the  estab¬ 
lishment  of  the  100  discharge  cycle  increment.  The  decrement 
value  could  be  derived  by  taking  20  percent  (100  cycles/500 
cycles)  of  the  current  Independent  Government  Cost  Estimate 
(IGCE)  for  a  battery.  Or,  a  more  conservative  basis  would  be 
20  percent  of  the  cost  for  existing  conformable  wearable  bat¬ 
teries  (CWBs)  in  inventory.  In  similar  fashion,  a  compatibility 
decrement  look  like  this: 

$xxx,xxx  compatibility  decrement  for  integration  of  the  existing 
Universal  Battery  Fast  Charger  (UBFC)  in  the  proposed  solution. 
$xxx,xxx  compatibility  decrement  for  integration  of  the  existing 
Solar  Power  Based  Charger  (SPBC)  in  the  proposed  solution. 

The  service  life  of  chargers  greatly  exceeds  the  service  life  of 
rechargeable  batteries.  All  things  being  equal,  it  would  save 
the  government  money  and  prevent  the  waste  of  these  char¬ 
gers  if  future  batteries  were  compatible.  Prorating  the  value 
of  existing  chargers  and  basing  the  charger  quantity  on  an 
appropriate  ratio  for  size  of  this  battery  requirement  led  to  a 
defensible  decrement  value  for  compatibility. 

Yes,  this  scenario  is  overly  simplistic.  But  LCC  should  be  part  of 
our  decision  making  in  source  selections  and  this  hypothetical 
scenario  illustrates  that  VATEP  may  just  be  the  mechanism  to 
revive  LCC  evaluations. 

The  author  can  be  contacted  at  scott.gilbreth@dau.mil. 
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As  technological  advancements  increasingly  render  once  cutting-edge 
capabilities  obsolete  in  just  a  few  years,  the  Army's  ability  to  maintain 
technological  overmatch,  and  ultimately  combat  overmatch,  is  inextrica¬ 
bly  bound  up  in  its  ability  to  innovate— and  more  important,  to  harness  in¬ 
novation  to  rapidly  develop  game-changing  technologies  and  capabilities. 

Former  Secretary  of  Defense  Ashton  Carter  emphasized  the  important  role  of  innovation  in  the  U.S.  Army's  ability 
to  deliver  the  most  technologically  advanced  capabilities  to  our  soldiers.  An  example  of  one  of  his  recent  initiatives 
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member  of  the  Army  Acquisition  Workforce  with  Level  III  certifications  in  Engineering  and  Test  and  Evaluation.  He  has  a  bachelor  of  science 
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to  advance  innovation  in  the  Army  was  the  establishment  of 
the  Defense  Innovation  Advisory  Board.  Chaired  by  Alphabet 
Inc.  Executive  Chairman  Eric  Schmidt,  the  Board  is  to  include 
up  to  12  members  who  have  successfully  led  large  private  and 
public  organizations  and  have  excelled  at  innovating— rapidly 
identifying  and  adopting  new  technology  concepts. 

Among  other  things,  the  Board  was  tasked  with  advising  the 
Department  of  Defense  (DoD)  on  key  areas  central  to  success¬ 
ful  innovation,  including  rapid  prototyping.  At  the  same  time, 
House  Armed  Services  Committee  Chairman  Mac  Thornberry, 
Texas  Republican,  continues  advocating  rapid  prototyping  and 
has  suggested  that  the  Services  have  the  budget  flexibility  to 
"experiment  with,  prototype,  and  rapidly  deploy  weapon  sys¬ 
tem  components  and  other  technologies  without  requiring 
those  programs  be  tied  to  an  existing  major  program." 

Delivering  Innovative  Capabilities  Quickly, 
Cost  Effectively 

In  the  Army  Acquisition  Corps,  rapid  prototyping  enables  the 
material  developer  to  support,  test  and  field  cutting-edge 
capabilities  to  soldiers  in  shorter  time.  Once  developed,  the 
prototype  product  may  then  be  refined  to  meet  very  specific 
user  needs  and  expectations,  thereby  serving  as  both  the  basis 
for  and  fruit  of  innovation. 

Specifically,  prototyping  provides  the  Army's  Research,  De¬ 
velopment  and  Acquisition  Communities  with  several  advan¬ 
tages.  First,  it  enables  designers  and  engineers  to  move  their 
concepts  beyond  the  virtual  visualization  offered  via  traditional 
computer-aided  design  (CAD)  models.  For  example,  a  project 
manager  in  the  Acquisition  Corps  would  have  the  benefit  of 
seeing,  firsthand,  the  "look  and  feel"  of  a  potential  capabil¬ 
ity  long  before  he  or  she  begins  the  acquisition  process.  At 
this  point,  design  adjustments  may  be  considered  and  critical 
changes  may  be  made  that  improve  the  capability  at  the  most 
cost-efficient  point  in  the  development  process. 

Specifically,  with  a  physical  model  in  hand,  its  form,  fit  and 
function  can  be  soldier  verified,  which  would  likely  result  in 
fewer  design  changes  and  thereby  save  money.  This  kind  of 
additive  and  iterative  manufacturing  saves  both  the  time  and 
cost  typically  associated  with  further  development  of  patterns, 
molds  and  metals. 

Moreover,  rapid  prototyping  reduces  waste  and  rework— the 
design  may  be  rapidly  adjusted,  incorporating  user  feedback 
and  emerging  requirements.  Design  flaws  are  more  readily 
recognized  and  efficiently  addressed  in  the  early  stages  of 
prototyping.  This  is  particularly  important  since  it  enables  the 
Army  to  quickly  ensure  that  soldiers  receive  properly  work¬ 
ing  capabilities  while  fully  meeting  capability  requirements. 
Finally,  building  a  prototype  enables  the  materiel  developer 
to  understand  the  intricacies  of  a  given  product  or  system 
design.  This  understanding  can  inform  the  entire  acquisition 
process  by  better  positioning  the  combat  developer  to  ac¬ 
curately  determine  performance  requirements  and  provide 


early  information  to  the  materiel  developer  on  systems  en¬ 
gineering  processes. 

In  summary,  rapid  prototyping  serves  several  critical  pur¬ 
poses  within  the  research  and  development  and  acquisition 
communities.  It  may  be  leveraged  to  inform  and  influence 
early  design  and  development  efforts  for  capabilities  under 
consideration,  as  well  as  those  that  have  been  fielded  and 
are  experiencing  design  issues  and/or  not  meeting  soldier 
needs.  As  such,  rapid  prototyping  helps  ensure  that  we  pro¬ 
vide  our  soldiers  with  the  very  best,  innovative  capabilities 
in  the  shortest  time  possible. 

Army  Rapid  Prototyping  Division 

The  Army's  Special  Products  and  Prototyping  Division  (SPPD) 
is  one  of  seven  divisions  within  the  Fort  Belvoir,  Virginia, 
Night  Vision  and  Electronic  Sensors  Directorate  (NVESD) 
of  the  Army's  Communications-Electronics  Research,  Devel¬ 
opment  and  Engineering  Center  (CERDEC).  One  of  SPPD's 
primary  core  competencies  is  rapid  prototyping  and  the  de¬ 
velopment  of  system  surrogates  and  components  to  enable 
the  Army  Research  and  Development  and  Acquisition  Com¬ 
munities  to  assess,  test  and  field  innovative  capabilities  to 
warfighters  very  quickly.  Rapid  prototyping  has  a  rich  history 
dating  back  to  the  early  1940s  when  the  very  first  facilities 
were  established  at  Fort  Belvoir.  Prototyping  activities  in 
those  days  focused  on  providing  support  for  war  efforts  by 
rapidly  building  systems  (and  components),  and  this  kind  of 
full  fabrication  service  continued  until  the  early  1990s  with 
the  facility's  fabrication  of  components  and  surrogates  for 
the  Patriot  Missile  System  and  mine  plows  in  support  of  Op¬ 
eration  Desert  Storm  in  1991.  By  rapidly  responding  to  sol¬ 
dier  concerns  in  facing  in-depth  obstacles,  the  shop  quickly 
designed  and  fabricated  mechanical  solutions  to  neutralize 
the  threat.  Then,  in  1993,  as  a  result  of  the  Defense  Base 
Closure  and  Realignment  Commission  findings,  the  Model 
Fabrication  Division  (and  its  mission)  became  the  NVESD 
Prototyping  Facility.  At  the  same  time,  the  facility  underwent 
a  transition  from  a  limited-production  service  model  to  a 
rapid  prototyping  service  model.  This  shift  in  mission  better 
positioned  the  facility  to  respond  to  emerging  soldier  needs 
and  to  provide  the  Research,  Development  and  Acquisition 
Communities  with  the  ability  to  rapidly  generate  prototypes 
that  could  be  used  to  inform  both  the  design  and  develop¬ 
ment  of  a  product,  and  later,  its  acquisition. 

By  the  late  1990s,  the  facility  was  fully  engaged  in  supporting 
the  Army  acquisition  community.  In  particular,  SPPD  forged 
a  strong  relationship  with  the  Project  Manager  Night  Vision/ 
Reconnaissance,  Surveillance,  and  Target  Acquisition  (now 
known  as  the  Project  Manager  Terrestrial  Sensors).  At  the 
time,  the  program  manager  (PM)  was  engaged  in  force  mod¬ 
ernization  efforts  and  "digitization"  of  the  battlefield.  In  partic¬ 
ular,  SPPD's  facility  supported  the  PM's  integration  of  sensors 
on  vehicles  by  mounting  them  via  masts  and  custom  brackets. 
These  mast-mounted  sensors  provided  critical  surveillance 
capabilities  in  support  of  the  Bosnia  and  Kosovo  missions. 


31 


Defense  AT&L:  May-June  2017 


Bill  Ruthenberg,  SPPD  mechanical  engineering  technician:  "I  work  extensively  with 
the  engineers  and  customers  during  the  design  phase,  advising  them  on  fabrication 
methodology  and  manufacturability  issues." 


After  the  September  11,  2001,  attacks, 
the  facility  provided  rapid  prototyping 
for  various  efforts  directly  support¬ 
ing  both  Operation  Iraqi  Freedom  and 
Operation  Enduring  Freedom  (OEF). 
Once  again,  the  facility  played  an  in¬ 
tegral  role  in  the  integration  of  critical 
reconnaissance  and  surveillance  sen¬ 
sors,  including  the  Driver's  Vision  En¬ 
hancer  and  the  Long  Range  Advanced 
Scout  Surveillance  System  (LRAS3)  on 
vehicles.  Particularly  during  the  early 
years  of  the  War  on  Terror,  each  time 
the  armor  changed  on  a  vehicle,  the  ve¬ 
hicle  required  a  new  sensor  integration, 
and  the  facility  was  able  to  step  in  and 
rapidly  conduct  the  integrations  for  the 
PM.  Moreover,  because  the  facility  was 
involved  in  each  integration,  the  ma¬ 
chinists  were  able  to  keep  pace  with  the 
various  configurations  and  anticipate 
the  next  likely  enhancement,  saving  the 
PM  time  and  delivering  support  to  sol¬ 
diers  faster  and  more  efficiently.  And, 
when  the  Army  shifted  to  the  use  of  the 
Mine  Resistant  Ambush  Protected  ve¬ 
hicle  in  response  to  the  enemy's  use  of 
improvised  explosive  devices,  the  facil¬ 
ity  was  ready  to  support  sensor  integra¬ 
tion  on  these  critical  vehicles. 

Partnering  With  SPPD 

Today,  the  facility  provides  rapid  proto¬ 
typing  support  to  various  Army  orga¬ 
nizations.  Troops  from  the  field  often 
come  to  the  facility  with  a  particular 
issue  or  need.  Unlike  the  facility's  in¬ 
ternal  and  regular  customers,  soldiers 
often  do  not  have  existing  relationships 
with  facility  technicians.  As  such,  the 
facility,  via  its  engineer  liaisons,  works 
closely  with  soldiers  to  understand  their 
needs  and  to  design  and  develop  proto¬ 
types  for  testing  and  use. 

More  often  than  not,  Soldiers,  like  most 
customers,  find  it  difficult  to  articulate 
a  need  without  first  seeing  an  example. 
To  address  the  need,  engineers  and  sci¬ 
entists  deploy  as  uniformed  and  civilian 
science  advisers  to  work  directly  with 
soldiers  in  the  field  via  the  Field  Assis¬ 
tance  in  Science  and  Technology  pro¬ 
gram.  They  review  soldier  descriptions 
of  needs  and  revise  them,  as  necessary, 
before  sending  them  to  and  conferring 
with  the  facility  technicians.  Engineers 


then  work  closely  with  the  customer  team  to  clearly  define 
the  issue,  develop  a  problem  set,  and  determine  options  to 
best  address  the  issue.  The  effort  is  collaborative,  with  facility 
members  providing  key  subject-matter  expertise  and  advice. 
Because  deployment  rotations  are  fairly  frequent,  problems 
and  issues  frequently  are  similar  to  those  the  facility  previ¬ 
ously  worked  on  even  up  to  10  to  15  years  earlier.  Several  of  the 
technicians  have  worked  at  the  facility  for  15  years  or  more,  so 
they  often  can  recall  the  very  solutions  they  developed  numer¬ 
ous  years  prior! 

Once  a  preliminary  solution  is  agreed  upon,  the  facility 
team  develops  the  prototype  and  delivers  it  to  the  user  for 
initial  use  and  assessment.  This  initial  use  and  assessment 
is  one  of  the  major  value-added  aspects  of  rapid  prototyp¬ 
ing.  The  user  is  able  to  "touch  and  feel"  the  product,  and, 
in  turn  determine  whether  it  meets  their  needs  or  requires 
further  development.  After  the  user  is  satisfied  with  the 
prototype,  the  user  then  decides  whether  to  pursue  funding 
to  produce  the  product.  When  funding  is  secured,  facility 
personnel  work  with  the  engineers  and  scientists  in  prepar¬ 
ing  the  necessary  documentation  to  begin  prototyping  once 
the  customer  approves  the  plan.  The  facility's  work  is  not 
finished  once  the  prototype  is  complete.  Technicians  often 
provide  assistance  and  training  for  the  product,  as  well  as 
field  support,  ensuring  that  the  soldier  can  realize  the  full 
potential  of  the  fielded  product. 


area  assembles  the  completed  items,  and  incorporates  them 
into  the  final  product,  system  or  platform  for  which  they  were 
intended.  This  integration  begins  with  the  design  phase  and 
carries  through  to  the  end  product. 

In  addition  to  the  conventional  techniques,  the  facility  utilizes 
3D  printing  or  additive  manufacturing  processes  to  fabricate 
prototype  components  that  can  be  readily  transitioned  to  pro¬ 
duction  processes  such  as  injection  molding  or  casting. 

Supporting  the  Nation 

Over  the  last  decade,  SPPD's  Prototyping  Facility  has  provided 
support  that  has  directly  and  significantly  contributed  to  U.S. 
efforts  both  at  home  and  abroad.  Its  aircraft  integration  ef¬ 
forts  include  design,  ensuring  compliance  with  Airworthiness 
Release  requirements,  crafting,  mounting  and  integration  of 
prototype  experimental  sensors  on  various  aircraft,  such  as  the 
Black  Hawk  helicopter.  These  sensors  aid  pilots  by  providing, 
among  other  things,  improved  visibility  in  degraded  conditions, 
thereby  enabling  safer  takeoff  and  landing. 

The  facility  also  provides  support  to  reset  and  retrograde 
sensor  systems  returning  from  theater.  At  the  height  of  the 
wars  in  Iraq  and  Afghanistan,  security  for  large  forward  op¬ 
erating  bases  became  increasingly  important.  Securing  entry 
points  and  enabling  situational  awareness  were  paramount 
for  warfighter  safety  and  survivability.  In  response,  scientists 


During  the  early  years  of  the  War  on  Terror, 
each  time  the  armor  changed  on  a  vehicle, 
the  vehicle  required  a  new  sensor  integration, 
and  the  facility  was  able  to  step  in  and  rapidly 
conduct  the  integrations  for  the  PM. 


The  facility  includes  five  main  areas:  design;  machining;  sheet 
metal  fabrication;  welding;  and  integration.  The  designers  in¬ 
terface  directly  with  users,  requirements  developers,  and  the 
technicians  to  generate  and  document  an  optimal  system  or 
component  design.  The  machining  area  utilizes  manual  and 
Computer-Numerical-Controlled  milling  and  turning  pro¬ 
cesses  to  fabricate  low-quantity  prototype  and  production- 
quality  components.  These  machining  processes  start  with 
large  pieces  of  metal  or  composites  and  remove  material  to 
achieve  the  final  size  and  dimensions.  The  sheet-metal  fabri¬ 
cation  area  constructs  mechanical  components,  enclosures 
and  panels  with  capabilities  to  shear,  bend,  punch  and  roll 
many  types  of  material.  The  welding  area  performs  all  com¬ 
mon  welding  processes  on  materials  from  thin  sheet  metal 
up  to  armor  plate  and  large  steel  weldments.  The  integration 


and  engineers  at  NVESD  rapidly  developed  key  elements  of 
the  Base  Expeditionary  Targeting  and  Surveillance  Systems— 
Combined  or  "BETSS-C"  a  family  of  sensor  systems  managed 
by  the  Army's  Project  Manager,  Terrestrial  Sensors  (PM  TS) 
and  intended  to  provide  base  defense  and  security  capabilities. 
Originally  intended  to  provide  perimeter  security,  BETSS-C's 
legacy  configuration  was  not  fully  adaptable  to  worldwide 
deployments,  including  expeditionary  missions.  In  response, 
SPPD  developed  and  supported  an  innovative  reset  concept 
for  PM  TS  that  was  intended  to  not  only  reset  the  existing 
BETSS-C  systems,  but  to  also  make  them  multi-mission  ca¬ 
pable,  thereby  infusing  the  system  with  increased  capabilities 
that  are  essential  for  succeeding  in  various  expeditionary  mis¬ 
sions  and  provide  the  warfighter  with  greater  flexibility  and 
more  value  from  a  single  sensor  system. 
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A  major  change  to  the  BETSS-C  family  of  systems  was  the 
upgrade  of  the  Cerberus  sensor  system.  Initially  fielded  in 
Iraq  in  2007,  the  Cerberus  is  a  tactical  trailer  with  a  tower 
sensor  system  that  provides  warfighters  with  surveillance 
and  reconnaissance  capabilities  via  a  radar  and  a  day/night 
sensor  mounted  on  a  readily  deployable  20-foot-high  tower. 
The  facility  worked  with  PM  TS  and  a  team  of  engineers  to 
design  the  components  and  to  build  quality  into  the  product 
at  the  earliest  stages  of  design.  Three  kits  were  developed  to 
upgrade  the  Cerberus  trailer,  thereby  improving  the  ease  with 
which  it  may  be  deployed  to  better  support  expeditionary  mis¬ 
sions.  The  facility  then  led  retrofit  activities  for  approximately 
100  Cerberus  trailers  by  developing  and  assembling  the  initial 
production  kits. 


proposal  and  awarding  a  contract  (to  defense  contractors)  for 
the  production  of  the  Mobile  Surveillance  System  (MSS).  The 
MSS  has  been  deployed  along  the  southern  border  in  direct 
support  of  CBP  efforts  and  has  proven  essential  for  helping  to 
secure  the  nation. 

The  facility  also  provides  support  within  NVESD.  When 
NVESD  needed  a  laboratory  specifically  designed  to  sup¬ 
port  calibration  efforts  for  various  hyperspectral  sensors, 
it  called  on  the  Prototyping  Facility  to  assist  with  designing 
and  fabricating  various  elements,  including  sensor  mounts  to 
conduct  calibration  and  staging  materials  to  move  a  sensor 
among  the  various  calibration  stations.  The  facility's  support 
enabled  NVESD  to  automatically  achieve  bore  sighting  at  all 


To  maintain  its  position  as  the  world's 
best  army,  the  U.S.  Army  must  innovate 
regularly,  rapidly  and  successfully.  Rapid 
prototyping  is  an  integral 
part  of  innovation. 


In  addition  to  supporting  military  efforts,  the  facility  supports 
humanitarian  efforts  around  the  world.  Led  by  a  team  within 
NVESD's  Countermine  Division,  the  Humanitarian  Demining 
effort  spans  several  countries  that  historically  have  suffered 
from  the  effects  of  years  of  war,  including  remaining  land 
mines  and  other  explosive  ordnance.  The  Countermine  Divi¬ 
sion  designs  the  systems  that  assist  these  countries  with  the 
identification  and  destruction  of  underground  hazards,  and  the 
Prototyping  Facility  supports  fabrication  for  systems,  spares, 
modifications  and  repairs.  Machinists  and  Countermine  spe¬ 
cialists  often  produce  new  vehicle  "skins,"  integrate  mine  re¬ 
moval  and  mine-detection  sensors  on  vehicles,  and  design  and 
produce  digging  devices  to  remove  ground  dirt  while  searching 
for  explosive  hazards.  Because  NVESD  designs,  evaluates  and 
fabricates  the  prototypes  exclusively  in-house,  it  is  able  to  rap¬ 
idly  deliver  to  countries  in  need  these  lifesaving  capabilities  at 
a  lower  cost  compared  to  traditional  methods. 

The  facility  also  supported  U.S.  homeland  security  efforts.  A 
major  project  was  the  development  of  the  long-range  variant 
of  the  Cerberus  tower  funded  by  the  Department  of  Homeland 
Security  (DHS).  The  facility,  working  with  design  engineers 
within  SPPD,  and  through  the  spiral  development  process, 
fabricated  several  prototype  variants  for  use  by  U.S.  Customs 
and  Border  Protection  (CBP)  personnel.  Based  on  testing  along 
the  southern  border  and  feedback  from  agents,  a  final  variant 
was  prototyped,  which  led  to  DHS  submitting  a  request  for 


sensor  calibration  stations  following  the  initial  bore  sighting. 
This  change  automates  the  process,  thereby  reducing  oppor¬ 
tunities  for  error  and  also  reducing  calibration  cycle  times  from 
1  week  to  1  day. 

To  maintain  its  position  as  the  world's  best  army,  the  U.S.  Army 
must  innovate  regularly,  rapidly  and  successfully.  Rapid  pro¬ 
totyping  is  an  integral  part  of  innovation.  At  the  same  time, 
rapid  prototyping  also  can  also  help  in  assessing  solutions  for 
a  given  Army  requirement. 

As  technology  advances  ever  faster,  it  will  become  increasingly 
difficult  to  maintain  technological  superiority  without  continu¬ 
ous  innovation.  SPPD's  rapid  prototyping  facility  is  positioned 
to  support  the  Army's  Research,  Development  and  Acquisi¬ 
tion  Communities  in  the  integration  of  the  Army's  innovation 
efforts  by  leveraging  engineering  and  technical  expertise  to 
inform  product  design,  development  and  transition  to  pro¬ 
curement.  This  support  enables  the  acquisition  community 
to  better  tailor  form,  fit  and  function  to  precise  soldier  needs. 
Harnessing  the  power  of  rapid  prototyping  will  better  position 
the  Army  to  discover  unique,  cutting-edge  solutions  to  today's 
problems  and  develop  game-changing  capabilities  that  will 
shape  the  battlefield  of  tomorrow. 


The  authors  can  be  contacted  at  richard.e.hott.civ@mail.mil  and 
christina.m.bates22.ctr@mail.mil. 
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Solving  Supply  Problems 


Tom  Gerbe 


Over  the  last  16  years,  staff  members  of  the  commercial  data  provider 
where  I  work  have  spoken  with  more  than  30,000  military  and  govern¬ 
ment  contractors.  This  experience  has  given  our  firm,  which  is  focused 
on  solving  supply  problems  for  the  U.S.  military,  a  unique  insight  into  the 
plight  of  people  seeking  products  and  services.  The  bulk  of  our  traffic 
can  be  separated  into  two  major  categories: 

■  Addressing  the  need  to  replace  a  part. 

■  Conducting  market  research  to  assemble  formal  procurement  or  justification  and  approval. 

Surprisingly,  many  items  sought  are  common,  everyday  things.  Wire  cutters,  fire  safety  equipment,  pulleys,  an¬ 
tennae,  screwdrivers  and  laptops  are  examples.  Of  course,  people  also  search  for  equipment  designed  to  meet 
needs  that  are  specific  to  the  military.  One  of  my  favorites  is:  "rocket  motor,  trajectory  divergence"  (National  Stock 
Number  [nsn]):  1377-01-256-1971),  which  is  used  in  ejector  seats.  We  recently  have  seen  a  lot  of  bid  activity  for 
FSC 1377,  cartridge  and  propellant  devices. 


Gerbe  is  a  defense  industry  analyst  for  BidLink.net  a  provider  of  information  to  defense  contractors  and  the  U.S.  military.  BidLink.net  provides 
tools  and  training  to  streamline  sourcing  and  procurement. 


The  job  of  a  program  manager 
(PM)  involves  managing  cost, 
schedule,  performance  and  risk. 
Efficient  acquisition  of  goods  and 
services  is  essential  to  maintain¬ 
ing  this  balance.  I  believe  that  the 
PM's  job  was  best  described  in  A 
Guide  for  DoD  Program  Managers 
written  by  William  T.  Cooley  and 
Brian  C.  Ruhm  and  published  by 
the  Defense  Acquisition  Univer¬ 
sity  in  2015: 

"Eliminating  or  correcting  root 
causes  that  will  otherwise  re¬ 
sult  in  perturbations  to  the 
cost,  schedule,  or  performance." 


There  was  some  controversy  surrounding  the  F-35  ejector  seat.  The  initial  ejector  system  put 
F-35  pilots  weighing  less  than  136  pounds  at  risk  of  neck  injury.  Martin  Baker  implemented  a 
three-point  plan  reducing  weight  in  the  helmets,  made  software  fixes  of  the  rockets'  timing,  and 
added  a  support  panel  to  the  headrest.  A  simpler,  one-point-plan  might  be  to  order  up  some 
Krispy  Kreme  donuts  for  those  skinny  pilots  (nsn:  8920-00-194-8601;  "cookies,  sandwich"). 


Time  and  time  again  we  see  peo¬ 
ple  without  the  proper  tools  who  are  trying  to  find  something. 
They  usually  find  us  through  a  Google  search  for  a  contract  or 
item  name,  indicating  that  they  have  exhausted  the  govern¬ 
ment-provided  tools. 

More  Than  Just  Contracting 

Buyers  and  contracting  officers  are  not  the  only  ones  search¬ 
ing  for  parts.  Often  it's  someone  who  needs  a  particular  part 
or  tool  in  order  to  complete  a  job.  At  the  Naval  Air  Station  in 
Jacksonville,  Florida,  we  met  with  a  fabricator  who  said  that 
he  spends  more  than  half  of  his  time  searching  for  things 
instead  of  fabricating.  Fie  was  trying  to  locate  a  specialized 
pulley  that  could  work  with  polyethylene  rope.  Lack  of  that 
pulley  was  preventing  him  from  completing  a  project.  Every¬ 
one  we  met  at  Jacksonville  had  at  least  one  item  she  or  he 
was  unable  to  source. 

Without  proper  tools  or  training,  people  can  spend  many  frus¬ 
trating  hours  doing  things  to  support  their  jobs.  This  directly 
affects  schedule  and  program  performance.  With  a  little  train¬ 
ing  and  the  right  tools,  PMs  can  locate  efficient  personnel  in 
many  departments  who  can  help  them  quickly  find  what  they 
need.  Perhaps  programs  could  offer  specialized  assistance  in 
locating  commercial  off-the-shelf  items  so  that  end  users  can 
spend  more  time  working  and  less  time  searching. 


be  seen  when  FPDS  information  is  compared  to  a  complete 
Defense  Logistics  Agency  (DLA)  procurement  history.  Com¬ 
mercial  data  providers  spend  time  filling  these  gaps,  as  well  as 
cleaning  out  errors  from  the  data.  DLA  often  moves  decimal 
places  when  unit  pricing  of  its  data  exports. 

Commercial  services  also  consolidate  the  data  from  many  gov¬ 
ernment  databases.  These  data  can  be  displayed  together  on 
a  single  page,  instead  of  having  to  search  a  half-dozen  data¬ 
bases  to  find  an  answer.  This  saves  time  and  helps  maintain 
program  schedules. 

Simple  Tips  for  Better  Sourcing 

We  will  cover  some  techniques  and  common  mistakes  that 
we  have  encountered  when  helping  people  source  parts  and 
services.  These  can  prove  helpful  for  managers  and  contract¬ 
ing  personnel,  as  well  as  end  users: 

Search  Simple:  Instead  of  searching,  "12-inch  steel  mounting 
bracket,"  start  with  the  word  "bracket."  Government  nomen¬ 
clature  is  in  reverse  order,  with  the  most  broad  category  first. 
A  mounting  bracket  is  categorized  as  "bracket,  mounting."  If 
there  is  no  specific  information  available  on  the  item  (i.e.,  NSN, 
part  number,  military  specification  [mil-spec]),  an  item  name 
search  for  "bracket"  is  a  good  starting  point. 


Choosing  the  Right  Tools 

The  government  provides  a  number  of  free  services  to  aid 
acquisition.  The  System  Acquisition  Management  database 
includes  every  defense  contractor  registered  to  sell  to  the 
government.  The  Federal  Logistics  Information  Service  (FLIS) 
Program  has  a  reference  system  for  parts  numbers.  The  Fed¬ 
eral  Procurement  Data  System  (FPDS),  although  incomplete, 
provides  a  basic  view  of  government  contracting  activity. 


Use  Mil-Specs:  The  government  maintains  a  free  database 
of  all  current  and  historical  military  specifications,  ASSIST, 
at  http://quicksearch.dla.mil/.  The  Master  Cross  Reference 
Database  (MCRD)  is  part  of  FLIS.  For  any  item  name  in  FLIS 
that  has  a  corresponding  mil-spec,  the  mil-spec  document  ID 
will  be  listed  in  the  part  number  cross  reference.  Sometimes 
mil-specs  include  part  numbers,  stock  numbers  and  similar 
items  that  can  be  a  good  starting  point  for  locating  a  source. 


On  the  other  hand  are  commercial  providers  who  add  value 
to  the  government  offerings.  For  example,  FPDS  data  are  re¬ 
ported  voluntarily  and  not  as  a  requirement.  Huge  gaps  can 


Prior  Solicitations:  Search  historical  solicitations  for  key¬ 
words  relating  to  the  item.  Not  all  agencies  release  detailed 
procurement  history.  Prior  solicitations  offer  more  detail  than 
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procurement  history  can  provide. 

Salient  characteristics  or  specifica¬ 
tion  numbers  are  included  in  the  so¬ 
licitation  and  can  help  locate  an  item. 

Colloquial  Names:  Common  names 
and  brand  names  may  not  yield  any 
results.  As  an  example,  a  client  was 
searching  for  "hard  hat"  and  could 
not  find  any  procurement  history. 

"Hard  hat"  is  a  popular  name  for  the 
DoD  nomenclature  "helmet,  safety." 

Proper  Fields:  It  may  seem  obvi¬ 
ous,  but  we  find  many  people  put  a 
part  number  in  the  NSN  field.  Even 
though  the  relevant  data  might  be 
in  our  system,  it  is  missed  due  to  a 
simple  error.  Try  to  utilize  multi-da¬ 
tabase,  intelligent  searches  when¬ 
ever  possible  to  avoid  this. 

GSAf  AbilityOne,  Unicor:  As  re¬ 
quired  by  Federal  Acquisition  Reg¬ 
ulation  (FAR)  8.4,  8.6,  and  8.7,  an  item  or  service  provided 
under  these  programs  should  be  sourced  through  the  afore¬ 
mentioned  channels.  We  have  a  number  of  commercial  clients 
with  thousands  of  items  in  their  catalogs  under  AbilityOne 
and  the  General  Services  Administration  or  GSA.  Because 
the  FAR  requires  the  use  of  these  programs  where  available, 
a  formal  procurement  history  may  be  lacking.  Always  check 
these  sources  before  buying. 

DoD  Email  and  Credit  Card:  Full  Text  search  DoD  Email  and 
credit  card  transactions  for  relevant  item  names  and  keywords. 
Many  items  that  fall  below  the  micro  purchase  threshold  will 
not  appear  in  the  DLA  history  if  they  are  purchased  through 
Email  or  by  use  of  a  card. 

Technical  Characteristics:  Once  you  have  narrowed  your 
search  to  a  specific  item  name,  check  the  technical  charac¬ 
teristics  for  item  details.  These  will  include  all  salient  charac¬ 
teristics,  which  may  include  dimensions,  materials,  next  higher 
assembly,  voltage  output,  color,  among  others.  The  Item  Name 
provides  a  broad,  general  description,  while  the  technical  char¬ 
acteristics  detail  the  item's  specific  properties. 

Procurement  History:  Conduct  a  key-word  search  of  the  all 
agencies'  history  to  locate  sources  that  previously  supplied 
the  item  to  the  military.  History  often  will  include  pricing 
information  in  order  to  support  cost  justification.  If  they  are 
available,  always  search  the  original,  signed  award  docu¬ 
ments  that  include  information  such  as  buyer  names,  contact 
information  and  shipping  destinations.  This  information  is 
omitted  from  the  feeds  available  directly  from  the  govern¬ 
ment.  Procurement  history  also  includes  the  contractor  Com¬ 
mercial  and  Government  Entity  (CAGE)  code,  which  can  be 


used  to  locate  a  vendor's  website  or 
contact  information. 

The  Contractor’s  Viewpoint 

Working  with  government  and  con¬ 
tractors,  we  get  to  see  both  sides  of 
the  procurement  process.  Contrac¬ 
tors  utilize  us  to  locate  solicitations 
and  associated  data— including 
technical  characteristics  and  past 
pricing.  We  regularly  speak  with 
staff  members  of  contractor  orga¬ 
nizations,  ranging  from  very  large 
firms  to  simple  two-person  shops. 
Below  are  a  few  issues  that  we  regu¬ 
larly  encounter: 

Better  Technical  Data:  Many  quali¬ 
fied  vendors  scour  the  Internet  for 
opportunities  to  sell  or  repair  parts 
to  the  military.  The  single  most  sig¬ 
nificant  factor  in  increasing  competi¬ 
tion  is  providing  vendors  with  quality 
technical  data  from  the  military.  PMs 
are  encouraged  to  participate  in  the  data  management  pro¬ 
cess.  I  cannot  stress  how  important  this  is  to  vendors  trying 
to  compete  on  contracts  for  obsolete  systems.  The  biggest 
complaint  from  contractors  is  the  lack  of  sufficient  documen¬ 
tation  on  systems. 

Access  to  Pricing  Data:  Some  agencies  will  not  release  line- 
item  pricing.  This  especially  seems  to  be  the  case  with  the 
U.S.  Navy  and  the  Army  Tank-Automotive  and  Armaments 
Command,  which  intentionally  omit  key  information  from 
all  Freedom  of  Information  Act  (FOIA)  requests— includ¬ 
ing  quantity  and  unit  price.  The  agencies  cite  Exemption  4 
of  FOIA  rules,  which  protects  "trade  secrets  and  commercial 
or  financial  information  obtained  from  a  person  [information 
that  is]  privileged  or  confidential."  There  has  been  an  ongoing 
battle  between  the  attorneys  for  large  contractors  and  the  U.S. 
military  regarding  this  information.  There  were  even  attempts 
to  clarify  the  issue  in  defense  authorization  bills,  to  require 
that  this  information  be  released.  To  date,  these  agencies  still 
omit  unit  price  and  quantity  from  their  responses  to  our  FOIA 
requests.  Both  contractors  and  military  need  to  know  what 
items  cost  in  order  to  work  effectively. 

Summary 

Supply  affects  every  part  of  a  program,  regardless  of  size.  From 
planning  through  support,  supply  is  a  key  aspect  of  program 
management.  Sourcing  is  not  limited  to  the  formal  contracting 
process,  as  people  throughout  all  areas  of  government  need 
to  locate  products  and  services  to  complete  their  job.  With 
the  proper  tools  and  training,  sourcing  can  be  more  effective, 
thereby  reducing  wasted  time.  & 

The  author  can  be  contacted  at  news@bidlink.net. 
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Welcome  to  the  new  DAU.mil 


WELCOME  TO  THE 
NEWWWW.DAU.MIL 


Our  updated  website 
features  improved  search 
features,  better  navigation 
and  newsharing  capabili¬ 
ties. 


We’ve  made  it  easier  to  get  the  informa¬ 
tion  and  news  you  need  from  the  ex¬ 
perts  in  one  convenient  location. 

When  you  search  DAU,  you  get  results 
from  ACQuipedia,  Communities  of  Prac¬ 
tice,  Defense  Acquisition  Portal,  Ask-A- 
Professor,  and  other  assets,  giving  you  a 
full  range  of  job  support  tools  when  you 
need  them. 

Our  new  website  offers  sharing  features 
that  encourage  collaboration.  In  our 


communities,  you  can  communicate 
directly  with  acquisition  professionals. 
When  you  follow  our  biogs  and  articles, 
you  will  be  automatically  updated  on  the 
latest  in  acquisition  news  and  trends. 

You  wanted  to  use  our  tools  on  the  go, 
so  we’ve  put  them  in  your  pocket.  Our 
updated  site  has  an  improved  mobile 
interface  and  that  gives  you  the  same 
capabilities  whether  at  the  desk  or  in 
the  field. 
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The  Department  of  Defense  (DoD)  finds  it  difficult  to  determine  when  higher 
technical  ratings  for  contractor  proposals  justify  paying  a  higher  price.  The 
acknowledged  difficulty  in  making  such  decisions  and  the  magnitude  of 
DoD  acquisitions  for  which  trade-off  decisions  are  needed  demand  resolu¬ 
tion  of  this  problem.  The  October  2010  Government  Accountability  Office 
(GAO)  report  GAO-1T8,  Enhanced  Training  Could  Strengthen  DOD's  Best  Value  Tradeoff 
Decisions,  emphasized  the  difficulty: 

According  to  DOD  officials,  making  sound  trade-off  decisions,  and  in  particular,  deciding  whether  or  not  a  price  differential 
is  warranted,  is  one  of  the  most  difficult  aspects  of  using  a  best  value  trade-off  process. 

The  magnitude  of  technical/cost  trade-off  decisions  is  evidenced  by  the  fact  that  best  value  process  is  applicable 
in  roughly  95  percent  of  DoD's  new,  competitively  awarded  contracts  valued  at  $25  million  or  more  as  reported 
in  that  GAO  report. 


Curry  is  president  of  WSC  Consulting  in  Chico ,  California.  He  is  the  author  of  second  edition  of  "Government  Contracting:  Promises  and 
Perils ,  2nd  Edition published  by  Routledge,  2017. 
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The  decision  matrices  prepared  by  the  source- 
selection  evaluation  board  (SSEB)  and  presented 
to  the  source-selection  authority  (SSA)  often  were 
cryptic  or  in  other  ways  failed  to  facilitate  technical/ 
cost  trade-off  decisions. 


DoD's  contractor  selection  process  is  described  by  an  April  1, 
2016,  memorandum  on  "Department  of  Defense  Source  Se¬ 
lection  Procedures,"  from  the  then  Under  Secretary  of  Defense 
for  Acquisition,  Technology,  and  Logistics  Frank  Kendall.  Use  of 
the  guidelines  in  this  memorandum  in  the  contractor  selection 
process  is  mandated  by  Defense  Federal  Acquisition  Regula¬ 
tion  Supplement  (DFARS)  PGI  215.3— Source  Selection.  The 
source-selection  procedures  in  the  2016  memorandum  are 
superior  to  and  provide  greater  detail  than  the  2011  memoran¬ 
dum.  Unfortunately,  however,  certain  deficiencies  in  the  2011 
memorandum,  were  carried  over  in  the  2016  memorandum, 
resulting  in  cryptic  contractor  selection  decision  matrices.  This 
article  examines  deficiencies  in  the  2016  memo's  method  for 
rating  contractor  proposals  and  describing  the  importance  of 
factors  and/or  subfactors.  And  it  pursues  recommendations 
for  making  improvements. 

One  approach  this  author  used  to  understand  weaknesses  in 
the  federal  contractor  selection  practices  involved  evaluating 
GAO  decisions  that  sustained  protests  from  Aug.  1,  2010,  to 
July  31, 2012.  The  GAO's  stated  reasons  for  sustaining  protests 
can  be  considered  as  weaknesses  in  the  contractor  selection 
process.  GAO-sustained  protests  during  2015  and  2016  were 
selected  randomly  to  verify  the  continuation  of  those  weak¬ 
nesses.  All  65  GAO-sustained  protests  during  those  2  years 
were  evaluated.  The  author  noted  one  pertinent  weakness  in 
DoD's  contractor  selection  process  that  GAO  did  not  men¬ 
tion:  The  decision  matrices  prepared  by  the  source-selection 
evaluation  board  (SSEB)  and  presented  to  the  source-selection 
authority  (SSA)  often  were  cryptic  or  in  other  ways  failed  to 
facilitate  technical/cost  trade-off  decisions.  The  SSA  is  sent 
a  narrative  of  the  SSEB's  proposal  evaluation  activities,  but 
the  decision  matrices  offer  an  overview  of  that  evaluation  and 
provide  the  SSA  with  meaningful  additional  insight. 

I  believe  that  the  cryptic  contractor  selection  matrices  are  due 
to  the  DoD  restriction  of  numerically  indicating  the  impor¬ 
tance  of  evaluation  factors  and  of  numerically  rating  contractor 
proposals.  The  2016  memo  does,  however,  allow  waivers  of 
the  prescribed  proposal  rating  method  for  technical  factors 
and  subfactors.  The  2016  memo  continues  the  2011  memo 
requirement  that  only  relative  terms  be  used  to  represent  the 
importance  of  factors  and/or  subfactors  considered  in  select¬ 
ing  contractors;  adjectival  or  color  schemes  may  be  used  to 
score  the  proposals.  Notably,  the  numerical  rating  of  factors 


and/or  subfactors  is  permitted  by  Federal  Acquisition  Regula¬ 
tions  (FAR)  Section  15.305(a),  and  numerical  representation 
of  the  importance  of  evaluation  factors  also  is  permitted  for 
Armed  Forces  contracting  by  10  U.S.  Code  §2305. 

Other  independent  research  into  GAO-reviewed  protests 
identified  a  way  to  evaluate  proposals  that  could  resolve  DoD's 
technical/cost  trade-off  dilemma.  That  2015  research  into  the 
best  state  and  local  government  contracting  practices  revealed 
how  to  clearly  identify  best  value  proposals.  The  findings  in¬ 
volved  15  states,  16  large  cities,  and  three  other  local  agencies 
and  was  a  follow-up  to  2006  research. 

Two  significant  errors  were  found  in  the  proposal  scoring  by 
the  majority  of  the  government  entities  reviewed  in  the  2015 
research.  But,  after  correction  of  those  errors,  their  approach 
to  selecting  contractors  was  found  to  be  superior  to  DoD's 
method  of  identifying  best  value  proposals.  The  errors  had  in¬ 
volved  using  anomalous  formulas  for  weighing  proposal  eval¬ 
uation  scores,  and  the  agencies  using  them  were  provided 
with  corrected  formulas.  Although  a  formal  follow-through 
was  not  deemed  appropriate,  a  separate  research  project 
noted  that  a  significant  number  of  government  entities  ad¬ 
opted  the  corrected  formulas.  With  that  correction  of  the 
formulas,  the  revamped  state  and  local  government  proposal 
evaluation  process  would  greatly  benefit  DoD's  technical/ 
cost  trade-off  identification  of  contractors  proposing  the  best 
values.  The  approach  recommended  here,  besides  better 
identifying  best  value,  will  guard  against  corruption  of  the 
contractor  selection  process. 


Table  1.  The  Difficulty  of  Technical/Cost 
Trade-off  Decisions 


Competing 

Contractors 

GID 

Patricio 

Management  and 
Staffing  Capability 

Marginal 

Outstanding 

Price 

$13,915,920 

$18,500,928 

Past 

Performance 

Substantial 

Confidence 

Satisfactory 

Confidence 
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Table  2.  Tendency  to  Tie  Scores  Despite  Discerned 
Proposal  Quality  Differences 


AS&D 

JTI 

Technical 

Scenario 

Outstanding 

Outstanding 

Program  Management 

Outstanding 

Outstanding 

Subcontract  Management 

Acceptable 

Acceptable 

Phase-In  Plan 

Outstanding 

Outstanding 

Past  Performance 

Substantial  Confidence 

Substantial  Confidence 

Proposed  Cost/Price 

$207,881,101 

$208,512,402 

Evaluated  Cost/Price 

$207,881,101 

$208,512,402 

Two  recent  examples  of  contractor 
selection  decision  matrices  that  ob¬ 
scure  identification  of  contractors 
submitting  best  value  proposals  are 
provided  below.  The  first  example 
(Table  1)  comes  from  GAO's  decision 
in  the  matter  of  Patricio  Enterprises, 
Inc.;  File  B-412740,  B-412740.3,  and 
B-412740.4,  May  26,  2016.  This  il¬ 
lustrates  the  difficulty  in  making 
technical/cost  trade-off  decisions. 


The  proposed  management  and 
staffing  capability  of  Patricio  Enter¬ 
prises  was  undeniably  superior  to 
its  competitor,  GID.  The  price  pro¬ 
posed  by  GID  and  the  GID  past  performance  rating  favored 
contract  award  to  GID.  The  solicitation  stated  that  the  evalu¬ 
ation  factors  (management  and  staffing  capability,  price,  and 
past  performance)  were  to  be  listed  in  descending  order  of 
importance.  In  making  the  contractor  selection  decision,  the 
SSA  could  justify  a  statement  that  the  superior  management 
and  staffing  capability  of  Patricio  outweighed  the  lower  pricing 
and  better  past  performance  of  GID.  But  the  SSA  could  just 
as  easily  have  stated  that  GID's  lower  price  and  better  past 
performance  outweighed  Patricio's  superior  management  and 
staffing  capability.  The  best  value  proposal  was  not  expressly 
identified  in  this  decision  matrix. 

The  other  recently  published  decision  (Table  2)  is  from  GAO's 
decision  in  the  matter  of  Jacobs  Technology,  Inc.  (JTI);  File 
B-413389  and  B-413389.2,  Oct.  18, 2016.  In  this  example  of  a 
sustained  protest,  the  decision  matrix  illustrated  the  tendency 
to  obtain  tied  scores  despite  discerned  differences  by  the  SSEB 
in  the  quality  of  the  proposals. 

Apart  from  the  proposed  and  evaluated  cost/price,  all  the 
ratings  reflected  in  Table  2  are  identical.  The  following  nar¬ 
rative  excerpted  from  the  GAO  decision,  however,  indicated 
differences  were  discerned  in  the  quality  of  numerous  factors: 


Table  3.  Decision  Matrix  Identifying  the  Best 


An  agency  source  selection  advisory  council  (SSAC)  then  con¬ 
ducted  a  comparative  assessment  of  the  offerors'  proposals. 
The  SSAC  found  that,  notwithstanding  the  equivalent  ratings, 
AS&D's  proposal  was  superiortothat  of  JTI  underthe  scenario, 
program  management,  and  phase-in  plan  subfactors  (the  offer¬ 
ors  were  considered  equal  underthe  subcontract  management 
subfactor).  ...  Similarly,  the  SSAC  found,  notwithstanding  the 
equal  ratings,  JTI's  past  performance  to  be  superior  in  relevance 
and  quality  to  that  of  AS&D. 

The  tendency  to  reflect  tied  scores  despite  factor  quality  dis¬ 
parities  should  be  disturbing  to  those  interested  in  the  avail¬ 
ability  of  an  effective  tool  for  making  intelligent  contractor 
selection  decisions. 

As  to  the  importance  of  the  evaluation  factors,  the  solicitation 
for  the  above  procurement  action  stated  the  following: 

The  solicitation  established  that  contract  award  would  be 
made  on  a  "best  value"  basis,  based  on  three  evaluation  fac¬ 
tors  generally  in  descending  order  of  importance:  technical  risk 
(hereinafter,  technical);  past  performance;  and  cost/price.  ... 
The  technical  factor  was  comprised  of  four  subfactors  in  de¬ 
scending  order  of  importance:  scenario;  program  management; 
subcontract  management;  and  phase-in  plan. ...  The  technical 

and  past  performance 
evaluation  factors, 
when  combined,  were 
significantly  more  im¬ 
portant  than  cost/ 
price. 


To  illustrate  a  decision 
matrix  decidedly  iden¬ 
tifying  the  best  value 
proposal,  the  author 
applied  his  best  efforts 
to  assign  numerical  val¬ 
ues  to  the  significance 
of  evaluation  factors 
based  on  the  relative 


Value  Proposal 


AS&D 

JTI 

Weight 

Raw 

Weighed 

Raw 

Weighed 

Technical 

Scenario 

20 

10 

20.00 

9 

18.00 

Program  Management 

17 

10 

17.00 

9 

15.30 

Subcontract  Management 

14 

5 

14.00 

5 

14.00 

Phase-in  Plan 

11 

10 

11.00 

9 

9.90 

Past  Performance 

23 

6 

19.71 

7 

23.00 

Proposed/Evaluated  Cost/Price 

15 

$207,881,101 

15.00 

$208,512,402 

14.95 

TOTAL  WEIGHED  SCORE 

96.71 

95.15 
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The  tendency  to  reflect  tied  scores  despite  factor  I 
quality  disparities  should  be  disturbing  to  those 
interested  in  the  availability  of  an  effective  tool  for 
making  intelligent  contractor  selection  decisions. 


ranking  provided  in  the  Request  for  Proposals  (RFPs).  Best 
efforts  were  also  used  to  assign  numerical  proposal  evaluation 
scores  based  on  the  comparative  relationship  of  factor  scores, 
as  described  in  GAO's  decision. 

These  best-  efforts  numerical  scores  are  used  in  the  illustration 
(Table  3)  to  show  how  using  them  can  avoid  tied  scores  by  per¬ 
mitting  evaluators  to  reflect  the  quality  differences  discerned 
between  competing  proposals.  Numerical  representations  of 
the  importance  of  proposal  evaluation  factors  and  numerical 
scoring  of  proposals  permit  weighing  of  proposal  evaluation 
scores  and  proposed  values  to  facilitate  unambiguous  identifi¬ 
cation  of  the  best-value  proposal.  The  formulas  used  to  weigh 
the  proposed  values  and  proposal  evaluation  scores  will  be 
described  later  in  this  article. 

In  the  Table  3  example,  AS&D,  with  the  highest  Total  Weighed 
Score  (TWS)  of  96.71,  is  identified  as  the  contractor  proposing 
the  best  value.  This  determination  is  based  on  the  author's 
best  efforts  to  quantify  DoD's  statement  regarding  the  factors' 
relative  importance,  the  contractor's  proposed  price,  and  the 
adjectival  proposal  evaluation  scores.  This  example,  however, 
merely  illustrates  the  superiority  of  the  TWS  approach.  The 
need  to  use  best  efforts  for  the  numerical  values  in  Table  3, 
does  not  support  the  argument  that  the  proposal  of  AS&D 
should  have  been  awarded  the  contract.  In  an  actual  procure¬ 
ment,  numerical  weights  and  the  numerical  scoring  scheme 
would  have  been  described  in  the  solicitation.  The  proposal 
evaluation  team  also  would  have  assigned  numerical  scores 
in  reviewing  the  proposals. 

The  TWS  approach  to  evaluating  proposals,  in  addition  to  dis¬ 
tinctly  identifying  the  best-value  proposal,  provides  the  needed 
transparency  so  that  prospective  contractors  can  understand 
the  precise  factor  weights  and  proposal  scoring.  The  transpar¬ 
ency,  moreover,  prevents  that  rare  corrupt  government  official 
from  manipulating  the  relative  value  of  proposal  evaluation 
factors/subfactors  to  justify  selecting  a  favored  contractor 
in  exchange  for  a  personal  benefit.  When  an  RFP  numerical 
representation  of  the  importance  of  contractor  selection  fac¬ 
tors,  that  corrupt  official  will  not  be  able  to  manipulate  factor 
weights  to  benefit  a  favored  contractor. 

The  statement  indicating  that  "technical  and  past  perfor¬ 
mance  evaluation  factors,  when  combined,  were  significantly 


more  important  than  cost/price"  injects  considerable  flex¬ 
ibility  for  concocting  and  rating  contractor  selection  factors/ 
subfactors.  The  factor  weight  for  cost/price  could  have  been 
within  the  range  of  10  to  25  and  still  have  been  considered 
significantly  less  important  than  the  combined  weight  of 
technical  and  past  performance  evaluation  factors.  The 
statement  regarding  the  relative  importance  of  the  nonprice 
factors  and  subfactors  also  permits  flexibility  in  manipulating 
the  relative  weights  during  proposal  evaluation  to  benefit  one 
of  the  competing  contractors. 

Formulas  for  Weighing  Proposal 
Evaluation  Scores 

The  anomalous  and  correct  formulas  for  weighing  proposal 
evaluation  rating  raw  scores  for  factors  such  as  technical  and 
past  performance  are  shown  below: 

Anomalous  Formula  for  Weighing  Proposal 
Evaluation  Scores 

Criterion  Weight  x  (Actual  Proposal  Evaluation  Score  ^  High¬ 
est  Possible  Proposal  Evaluation  Score)  =  Weighed  Score 

Correct  Formula  lor  Weighing  Proposal 
Evaluation  Scores 

Criterion  Weight  x  (Actual  Proposal  Evaluation  Score  -r  the 
Highest  Actual  Proposal  Evaluation  Score)  =  Weighed  Score 

The  difference  between  the  anomalous  and  the  correct  for¬ 
mula  for  weighing  proposal  evaluation  scores  involves  using 
either  the  highest  possible  or  the  highest  actual  proposal  evalu¬ 
ation  score.  The  necessity  of  adding  the  weighed  scores  for 
both  the  proposal  evaluation  scores  and  the  proposed  values 
requires  compatibility  between  their  respective  formulas.  The 
formula  for  weighing  proposed  value  scores  must  include  the 
highest  actual  proposed  value  because  there  can  be  no  highest 
possible  proposed  value.  Using  the  highest  possible  proposal 
evaluation  score  in  the  formula,  therefore,  distorts  the  algo¬ 
rithm  for  calculating  the  TWS.  Use  of  the  anomalous  formula 
may  result  in  an  understatement  of  the  weighed  value  for  pro¬ 
posal  evaluation  scores. 

The  problem  with  an  understated  weighed  value  for  a  pro¬ 
posal  evaluation  score  is  illustrated  in  the  following  example 
where  the  experience  and  price  factors  are  both  weighted 
as  10.  Evaluation  of  the  proposal  with  the  highest  score  for  a 
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technical  factor  (experience)  results  in  a  score  of  85  on  a  scale 
of  70-100.  When  the  anomalous  formula  is  used  to  weigh  the 
score  of  85,  the  result  is  8.5.  When  the  lowest  price  is  weighed 
with  the  formula  for  proposed  values,  the  result  is  10.0.  The 
weighed  score  of  8.5  for  the  highest  rated  experience  factor 
is  clearly  understated  when  compared  to  the  weighed  score 
of  10.0  for  the  lowest  price.  When  both  factors  are  weighted 
the  same,  the  highest  rated  experience  factor  and  the  lowest 
price  factor  should  have  identical  weighed  scores. 

Government  personnel  more  concerned  with  technical  perfor¬ 
mance  and  less  so  with  price  would  have  a  legitimate  reason  to 
object  to  the  calculation  of  weighed  scores  with  the  anomalous 
formula.  Technical  personnel  are  likely  to  object  to  a  weighed 
score  for  the  lowest  price  that  is  higher  than  the  weighed  score 
for  the  highest-rated  technical  factor  when  both  factors  have 
equal  weights.  When  applying  the  correct  formula,  the  low¬ 
est  price  and  highest-rated  technical  factor  will  have  identical 
scores  when  the  factor  weights  are  equal. 

Formulas  for  Weighing  Proposed  Values 

Formula  A  and  Formula  B,  below,  represent  two  differing 
methods  that  state  and  local  government  agencies  express 
the  anomalous  formula  for  weighing  proposed  values. 

Anomalous  Formula  A 

Lowest  Proposed  Value  -r  (Proposed  Value  ^  Criterion  Weight) 
=  Weighed  Score 

Anomalous  Formula  B 

(Lowest  Proposed  Value  -r  Proposed  Value)  -r  Criterion  Weight 
=  Weighed  Score 

Correct  Formula 

(Highest  Proposed  Value  -  (Proposed  Value  -  Lowest  Proposed 
Value))  -r  (Highest  Proposed  Value  ^  Criterion  Weight)  = 
Weighed  Score 

The  purpose  of  this  formula  is  to  weigh  proposed  values  ac¬ 
cording  to  the  factor  weight  and  convert  low  proposed  val¬ 
ues  to  high  weighed  scores.  This  approach  is  appropriate  for 
factors,  such  as  price,  where  low  values  are  favorable  to  the 
government.  Lower  proposed  values  are  also  favorable  to  the 
government  for  factors  such  as  "weight”  for  products  that  will 
be  placed  in  space  orbit.  The  formula  anomaly  is  character¬ 
ized  by  an  underrepresentation  of  midlevel  values  when  three 
or  more  contractors  compete  for  a  contract.  The  anomaly  is 
best  demonstrated  by  considering  three  proposed  equidistant 
prices.  In  this  example,  the  equidistant  prices  are  $700  mil¬ 
lion,  $800  million  and  $900  million,  with  an  equal  difference 
of  $100  million  between  the  low  and  midlevel  prices  as  well 
as  between  the  midlevel  and  the  highest  prices.  The  factor 
weight  in  this  example  is  30.  When  equidistant  proposed 
prices  are  weighed  and  low  prices  are  converted  to  high  scores, 
the  weighed  values  should  also  be  equidistant.  Weighing  the 
above  proposed  prices  with  the  anomalous  formula  yields  the 
following  results: 


Results  With  Anomalous  Formula 

Proposed  Value  Weighed  Score 

$700  Million  30.00 

$800  Million  26.25 

$900  Million  23.33 

The  weighed  scores  with  the  anomalous  formula  are  not  equi¬ 
distant  because  the  difference  in  the  weighed  scores  for  $700 
million  and  $800  million  is  3.75  while  the  difference  between 
the  weighed  scores  for  $800  million  and  $900  million  is  2.92. 

Results  With  Correct  Formula 

Proposed  Value  Weighed  Score 

$700  Million  30.00 

$800  Million  26.67 

$900  Million  23.33 

The  weighed  scores  with  the  correct  formula  are  equidistant 
because  the  difference  in  the  weighed  scores  for  $700  mil¬ 
lion  and  $800  million  is  3.33  and  the  difference  between  the 
weighed  scores  for  $800  million  and  $900  million  is  3.34. 
The  .01  difference  between  3.33  and  3.34  is  a  rounding  error. 

When  using  the  anomalous  formula  to  weigh  proposed  values 
where  low  numbers  favor  the  government,  the  weighed  scores 
for  the  highest  and  lowest  proposed  values  will  be  accurate. 
The  weighed  score  for  the  midpoint  values,  however,  will  be 
understated.  The  TWS  will,  therefore,  be  inaccurate  if  using 
an  anomalous  formula  thus  subjecting  the  government  to  the 
possibility  of  selecting  a  contractor  other  than  the  one  offering 
the  best  value. 

Conclusion 

The  author's  contention  is  that  the  difficulty  DoD  acknowl¬ 
edges  in  making  technical/cost  trade-off  decisions  results 
primarily  from  its  restrictions  against  numerical  represen¬ 
tations  for  the  importance  of  factors  and  subfactors,  and 
against  numerical  rating  of  contractor  proposals.  State  and 
local  government  agencies  often  represent  the  importance 
of  proposal  evaluation  factors  and  rate  proposals  numerically. 
These  government  contracting  agencies  use  formulas  to  weigh 
the  proposal  evaluation  ratings,  as  well  as  the  proposed  val¬ 
ues  such  as  price,  to  obtain  a  TWS  representing  the  score  for 
and  the  importance  of  each  evaluation  factor/subfactor.  The 
formula  for  weighing  price,  and  other  proposed  values  where 
low  values  are  favorable  to  the  government,  also  converts  low 
proposed  values  to  high  scores.  When  weighed  scores  for  fac¬ 
tors  and/or  subfactors  are  totaled,  the  result  is  a  TWS.  The 
contractor  receiving  the  highest  total  numerical  score  is  identi¬ 
fied  as  having  submitted  the  best  value  proposal. 

DoD  would  benefit  from  use  of  the  TWS  contractor  selection 
process  that  distinctly  identifies  the  contractor  offering  the 
best  value  proposal,  simplifies  the  technical/cost  trade-off 
decision,  and  inhibits  procurement  corruption.  & 


The  author  can  be  contacted  at  bcurry@wsc-consulting.com. 
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Future-Proof  Your  Program 


Roy  Wood ,  Ph.D. 


Things  change.  Stuff  happens.  Your  once  indomitable  program  is  now  being 
questioned  in  your  Service,  the  Office  of  the  Secretary  of  Defense  (OSD), 
or  Congress.  You  fear  your  program  may  be  on  next  fiscal  year's  chop¬ 
ping  block.  But  you  are  meeting  your  requirements— under  budget  and  on 
schedule.  What  could  possibly  have  gone  wrong? 

The  answer  is  that  your  program  may  not  have  been  adequately  future-proofed.  What  looked  good  5  or  10  years  ago 
to  sponsors  and  acquisition  executives  is  no  longer  so  shiny  and  valuable.  One  of  three  things  probably  happened. 

The  threat  changed.  We  rode  the  operational  roller  coaster  from  the  early  1990s  and  the  fall  of  the  former  Soviet 
Union  to  the  "long  war”  against  terrorism,  and  now  back  to  concerns  about  the  emergence  of  peer-competitors. 
Warfighting  systems  designed  for  limited  missions  or  specific  theaters  may  now  be  questioned  as  to  their  ability 
to  perform  in  the  perceived  current  threat  environment. 


Wood  is  a  former  program  manager  and  Defense  Department  executive.  He  currently  is  the  dean  at  Northeastern  State  University 
in  Broken  Arrow ,  Oklahoma. 
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So,  what  is  a  program  manager  (PM)  to  do?Jjow 
Lcana  program  be future-proofed^igainst threat 


The  technology  changed.  Technology  rapidly  changes,  and  we 
have  all  heard  how  difficult  it  is  to  synchronize  short,  frenetic 
technology  cycles  with  the  longer,  more  deliberate  acquisition 
cycle.  Systems  with  older  technologies  may  seem  out-of-date 
and  not  as  capable. 

The  people  changed.  Political  administrations  change.  Mili¬ 
tary  leaders  rotate  frequently.  Key  decision  makers  move  into 
and  out  of  influential  government  positions.  New  ideas  and 
competitors  emerge  who  purport  to  have  solutions  that  are 
better,  faster  and  cheaper  than  yours— and  can  prove  it  with 
two  dozen  or  so  animated  PowerPoint  slides. 

So,  what  is  a  program  manager  (PM)  to  do?  How  can  a  pro¬ 
gram  be  future-proofed  against  threat  and  technology  changes 
and  new  faces  who  may  not  be  proponents?  Let's  see  ... . 

The  Threat  Changed 

The  PM  clearly  has  little  say  in  how  the  threat  changes.  Poten¬ 
tial  adversaries  rarely  seek  our  advice.  So  the  PM  must  keep  an 
ear  to  the  ground  and  an  eye  on  the  horizon  for  threat  changes 
that  may  negate  program  effectiveness.  Here  are  some  ways 
to  do  that: 

■  Seek  out  intelligence  and  counterintelligence  briefings 
on  the  specific  threats  your  program  is  designed  to  thwart. 
Make  sure  your  intelligence  counterparts  know  enough 
about  your  program  that  they  can  be  on  the  lookout  for 
changes  in  the  threat  capability  that  may  impact  your  sys¬ 
tem's  effectiveness. 

■  Engage  with  your  operational  counterparts  or  system  spon¬ 
sors  to  get  their  advice.  Get  feedback  from  the  field  if  your 
system  is  already  in  the  hands  of  the  warfighter.  Look  for  ways 
to  make  your  system  more  valuable  to  the  users  based  on 
their  feedback.  Fans  in  the  field  can  go  a  long  way  to  ensuring 
support  for  your  program.  If  the  threat  has  evolved  beyond 
your  original  requirements  set,  enlist  your  sponsor's  help  in 
updating  the  requirements  documents.  Changes  in  these 
documents  may  be  the  leverage  you  need  to  make  necessary 
systems  changes. 


■  Use  evolutionary  acquisition  practices.  Fielding  a  system  as 
quickly  as  possible  will  often  get  a  capability  into  the  hands  of 
the  warfighter  before  threats  change.  Then,  intentionally  laying 
in  planned  upgrades  to  future  block  purchases  will  allow  the 
system  to  pace  the  threat  over  its  lifetime.  Don't  forget  to  also 
plan  for  back-fit  of  the  new  block  capability  into  earlier,  fielded 
versions  of  the  system. 

The  Technology  Changed 

Commercial  technology,  particularly  in  computing,  control, 
and  advanced  manufacturing,  is  moving  ahead  at  blazing 
speed.  Could  your  system  benefit  from  technologies  of  au¬ 
tonomous  vehicle  control?  How  about  3-D  printing?  Here  are 
some  ways  to  enable  this: 

■  Connect  with  your  science  and  technology  (S&T)  colleagues. 
Do  you  have  a  conduit  to  the  right  S&T  organizations  to  be 
able  to  incorporate  new  capabilities  into  your  system?  Make 
a  deliberate  effort  to  meet  with  your  Services'  S&T  leaders  to 
describe  your  program  and  seek  their  advice  and  assistance 
in  lining  up  potential  future  upgrades.  Also  consider  seeking 
help  from  OSD,  which  may  have  insights  into  applicable  tech¬ 
nologies  being  investigated  by  all  the  Services,  the  Defense 
Advanced  Research  Projects  Agency,  and  the  Defense  Innova¬ 
tion  Unit  Experimental— and  point  you  to  the  help  you  need. 

■  Work  with  your  prime  contractor  on  ideas  for  pre-planned 
product  improvements  and  build  these  into  your  extended 
program  and  contracting  plans. 

■  Use  Broad  Agency  Announcements  and  Requests  for 
Information  to  canvass  ideas  from  the  broader  technology 
community. 

■  Again,  be  sure  you  have  design  margin  to  support  new 
technologies.  Is  your  platform's  electrical  plant  capable  of 
powering  your  high-energy  laser  or  rail-gun?  Does  your  cool¬ 
ing  system  support  an  advanced  radar  upgrade?  Do  you  have 
enough  space  and  weight  capability  to  carry  a  larger  missile?  If 
not,  what  alternative  designs,  such  as  self-contained  systems 
with  their  own  power  and  cooling,  could  be  proposed? 


■  Make  sure  your  system  has  design  margin.  New  threats 
may  require  new  sensors  or  weapons.  Does  your  platform 
have  sufficient  design  margin  to  accommodate  upgrades? 
This  may  include  space  and  weight,  electrical  power,  and 
cooling  capability.  Design  margin  will  increase  your  cost, 
but  will  help  future-proof  your  system  if  and  when  the  threat 
changes. 


■  Finally,  technology  insertion  into  a  program  of  record  can 
best  be  accomplished  through  open  architecture  and  an  evo¬ 
lutionary  acquisition  program.  Block  upgrades  with  technology 
on-ramps  provide  a  timeframe  where  new,  sufficiently  mature 
technologies  can  be  inserted  without  disrupting  current  pro¬ 
duction.  Open  architecture  can  make  system  changes  easier 
and  cheaper. 
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The  People  Changed 

Given  the  frequent  turnover  of  senior  leaders  in  the  Pentagon, 
it  is  admittedly  difficult  to  keep  the  leaders  informed  about 
your  program  and  its  importance  to  national  security.  Adding 
to  the  challenge,  of  course,  are  potential  competitors,  whose 
ideas  and  programs  seem  to  ooze  into  the  building  armed  with 
flashy  presentations  and  promises  of  better,  faster,  cheaper. 
How,  then  is  a  PM  to  deal  with  this?  Here  are  a  few  ideas: 

■  Be  proactive.  Keep  your  current  sponsor  informed  and,  when 
that  sponsor  moves  on,  get  on  their  replacement's  calendar 
to  bring  him  or  her  up  to  speed.  Focus  on  why  the  program 
was  created  and  what  new  capabilities  it  offers  the  warfighter. 
Engage  the  sponsor  in  helping  you  keep  an  eye  on  the  evolving 
threat,  new  technologies,  and  new  people  he  or  she  believes 
you  should  also  brief. 

■  Set  up  a  stakeholder  "advisory  council."  Gather  your  key 
stakeholders  and  set  up  routine  opportunities  for  them  to  en¬ 
gage  in  program  update  meetings.  With  all  your  stakehold¬ 
ers  in  the  same  room,  many  sticky,  cross-cutting  issues  and 
problems  (resolution  of  which  might  otherwise  take  weeks  of 
round-robin  briefings)  can  be  dealt  with  during  the  meeting. 
Stakeholders  at  your  level  who  intimately  understand  your 
program  can  be  advocates  in  their  home  organizations  and 
provide  you  access  to  their  executives  if  the  need  arises.  This 
network  can  also  alert  you  to  potential  competing  ideas  and 
proposals  that  have  come  in  through  their  offices  so  you  can 
be  better  prepared  to  address  any  opposing  claims. 

■  Tell  the  truth,  never  equivocate— and  always  tell  a  consistent 
story.  Engage  your  stakeholders  proactively  and  truthfully.  If 
you  are  having  issues,  be  upfront  and  let  them  know.  They  may 
be  able  to  help.  At  least  they  will  feel  engaged  and  informed.  If 
you  are  battling  a  competing  program,  technology  or  idea,  do 
your  own  analysis  and  fairly  compare  the  alternative's  claims 
to  your  program's  reality.  If  the  other  idea  has  merit,  look  for 
ways  to  incorporate  it;  if  it  is  flawed,  help  objectively  and  dis¬ 
passionately  to  point  out  its  shortcomings  and  risks.  If  you 
have  inherited  a  program  that  truly  is  a  dog-that-won't-hunt, 
inform  your  chain  of  command  and  advocate  for  changes,  re¬ 
structuring,  or  even  cancellation.  Your  professional  integrity 
will  be  bolstered  and  both  the  warfighter  and  taxpayer  will  be 
better  served. 

Summary 

If  you  believe  that  a  PM's  job  is  to  remain  narrowly  focused 
on  managing  cost,  schedule  and  scope,  you  are  likely  to  miss 
the  threats  and  opportunities  coming  from  outside  your  pro¬ 
gram.  Whether  changes  to  the  external  environment  come 
from  potential  adversaries,  rapidly  evolving  technologies,  or 
new  decision  makers,  you  must  be  prepared.  To  do  so,  you 
need  to  raise  your  awareness  of  the  things  going  on  outside 
your  program  and  be  prepared  to  address  them  appropriately. 
Good  luck.  Godspeed. 

The  author  can  be  contacted  at  wood40@nsuok.edu. 
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